Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                

Introducing User Defined Functions preview for mapping data flows in ADF and Synapse

Published May 23 2022 10:57 AM 1,644 Views
Microsoft

We are excited to launch the public preview of user defined functions, a new feature for mapping data flows for Azure Synapse Analytics and Azure Data Factory. User defined functions allow users to define reusable functions that can be used across many mapping data flows. If you ever find yourself doing the same string manipulation, math calculations, or other complex logic several times this is a great use case for turning that into a user defined function.

 

User defined functions exist inside of a data flow library, this is a grouping of functions for whatever makes sense for your organization. This grouping could be based on related type of functions, business unit, or any other logical grouping that work for you. You can have multiple data flow libraries.

 

 

To get started with user defined functions, you must first create a data flow library. Navigate to the management page and then find data flow libraries under the author section.

 
 

After you create your data flow library you can add your user defined functions and can even add multiple arguments to make your functions more reusable.

 

data-flow-udf-function-pane

 

After you add and save your data flow library and user defined functions you can then access these in mapping data flows anywhere you can use the expression builder.

data-flow-udf-expression-builder

 

Along with user defined functions, the reusable Flowlets feature allows for multiple tools to create reusable logic that can be shared across many transformations and patterns in mapping data flows. We are excited to see what type of patterns you will all be building!

%3CLINGO-SUB%20id%3D%22lingo-sub-3414628%22%20slang%3D%22en-US%22%3EIntroducing%20User%20Defined%20Functions%20preview%20for%20mapping%20data%20flows%20in%20ADF%20and%20Synapse%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-3414628%22%20slang%3D%22en-US%22%3E%3CP%3EWe%20are%20excited%20to%20launch%20the%20public%20preview%20of%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fdata-factory%2Fconcepts-data-flow-udf%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3Euser%20defined%20functions%3C%2FA%3E%2C%20a%20new%20feature%20for%20mapping%20data%20flows%20for%20Azure%20Synapse%20Analytics%20and%20Azure%20Data%20Factory.%20User%20defined%20functions%20allow%20users%20to%20define%20reusable%20functions%20that%20can%20be%20used%20across%20many%20mapping%20data%20flows.%20If%20you%20ever%20find%20yourself%20doing%20the%20same%20string%20manipulation%2C%20math%20calculations%2C%20or%20other%20complex%20logic%20several%20times%20this%20is%20a%20great%20use%20case%20for%20turning%20that%20into%20a%20user%20defined%20function.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EUser%20defined%20functions%20exist%20inside%20of%20a%20data%20flow%20library%2C%20this%20is%20a%20grouping%20of%20functions%20for%20whatever%20makes%20sense%20for%20your%20organization.%20This%20grouping%20could%20be%20based%20on%20related%20type%20of%20functions%2C%20business%20unit%2C%20or%20any%20other%20logical%20grouping%20that%20work%20for%20you.%20You%20can%20have%20multiple%20data%20flow%20libraries.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%3ETo%20get%20started%20with%20user%20defined%20functions%2C%20you%20must%20first%20create%20a%20data%20flow%20library.%20Navigate%20to%20the%20management%20page%20and%20then%20find%20data%20flow%20libraries%20under%20the%20author%20section.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CDIV%20id%3D%22tinyMceEditorJoshuhaOwen_0%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CDIV%20id%3D%22tinyMceEditorJoshuhaOwen_1%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CP%3EAfter%20you%20create%20your%20data%20flow%20library%20you%20can%20add%20your%20user%20defined%20functions%20and%20can%20even%20add%20multiple%20arguments%20to%20make%20your%20functions%20more%20reusable.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22data-flow-udf-function-pane%22%20style%3D%22width%3A%20912px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F373698iC67A64B9C9C25AE8%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22data-flow-udf-function-pane%22%20alt%3D%22data-flow-udf-function-pane%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAfter%20you%20add%20and%20save%20your%20data%20flow%20library%20and%20user%20defined%20functions%20you%20can%20then%20access%20these%20in%20mapping%20data%20flows%20anywhere%20you%20can%20use%20the%20expression%20builder.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22data-flow-udf-expression-builder%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F373699i1960B6CBA381F596%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22data-flow-udf-expression-builder%22%20alt%3D%22data-flow-udf-expression-builder%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAlong%20with%20user%20defined%20functions%2C%20the%20reusable%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fdata-factory%2Fconcepts-data-flow-flowlet%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3EFlowlets%3C%2FA%3E%20feature%20allows%20for%20multiple%20tools%20to%20create%20reusable%20logic%20that%20can%20be%20shared%20across%20multiple%20many%20transformations%20and%20patterns%20in%20mapping%20data%20flows.%20We%20are%20excited%20to%20see%20what%20type%20of%20patterns%20you%20will%20all%20be%20building!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-3414628%22%20slang%3D%22en-US%22%3E%3CP%3EMapping%20data%20flows%2C%20in%20Azure%20Data%20Factory%20and%20Synapse%20Analytics%2C%20is%20the%20scale-out%20data%20transformation%20feature%20that%20allows%20data%20engineers%20to%20build%20low-code%20ETL%20jobs%20visually%20and%20execute%20using%20serverless%20Spark%20compute.%20Now%20with%20User%20Defined%20Functions%20(preview)%2C%20you%20can%20quickly%20build%20reusable%20function%20logic%20to%20share%20across%20multiple%20mapping%20data%20flows.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-3414628%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Factory%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMapping%20Data%20Flows%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20Synapse%20Analytics%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Co-Authors
Version history
Last update:
‎May 24 2022 12:15 PM
Updated by: