Azure Data flow: Flowlets for any input

Question

Azure Data flow: Flowlets for any input

Jan 1

I successfully created a flowlet for any input that trims down the amount of columns to a representative set. However when using the flowlet, the original projection of the input columns gets lost which complicates the implementation of the data flow, since there is no access to the mapping anymore.

For any kind of input, for example structered like this:

The output schema always looks like this (empty list):

The flowlet used does not remove or alter any of the existing columns and the returned output shown in the data preview also confirms this, so the original projection of the input columns should in theory still work. Is there any way to preserve the mapping without creating a flowlet for every specific case (registering all the inputs of the data flow)? Or is this something that might be resolved in a future update?

ShaikMaheer-MSFT 38,556 Reputation points Microsoft Employee Moderator

2022-11-29T13:52:05.94+00:00

Hi @Jan ,

Thanks for posting query in Microsoft Q&A Platform.

Little unclear with the ask actually. Unable to understand what you mean when says reated a flowlet for any input that trims down the amount of columns to a representative set.

Could you please share your flow let implementation details along with screenshots and usage of it in dataflow? So that I can try to repro same and understand better to help. Thank you.
Jan 1 Reputation point

2022-12-02T10:05:44.233+00:00

This goes for any flowlet that doesn't have specific input columns. For example this simple Flowlet

Which takes no specific input columns (1), creates a derived column based on the random function (2) and then sorts according to the new derived column (3) before returning the output.

When using the flowlet with an input like this

All it shows on the output is the newly created column (from step (2) in the flowlet).

This means that all columns that were used as the input can only be accessed via functions like byName, etc. Thus the originally mapped input columns won't show up anymore for simple access. For example when using sort it will only suggest the newly created column and all the mapped inputs from the source go missing.

I hope you can understand what I mean now.

1 answer

Your answer

ShaikMaheer-MSFT 38,556 Reputation points Microsoft Employee Moderator

2022-11-29T13:52:05.94+00:00

Hi @Jan ,

Thanks for posting query in Microsoft Q&A Platform.

Little unclear with the ask actually. Unable to understand what you mean when says reated a flowlet for any input that trims down the amount of columns to a representative set.

Could you please share your flow let implementation details along with screenshots and usage of it in dataflow? So that I can try to repro same and understand better to help. Thank you.
Jan 1 Reputation point

2022-12-02T10:05:44.233+00:00

This goes for any flowlet that doesn't have specific input columns. For example this simple Flowlet

Which takes no specific input columns (1), creates a derived column based on the random function (2) and then sorts according to the new derived column (3) before returning the output.

When using the flowlet with an input like this

All it shows on the output is the newly created column (from step (2) in the flowlet).

This means that all columns that were used as the input can only be accessed via functions like byName, etc. Thus the originally mapped input columns won't show up anymore for simple access. For example when using sort it will only suggest the newly created column and all the mapped inputs from the source go missing.

I hope you can understand what I mean now.

Answer 1

ShaikMaheer-MSFT 38,556 Microsoft Employee Moderator

Hi @Jan ,

Thank you for sharing details.

yes, it's expected behavior, let me explain in simple words, Flowlet is like a function which take some columns as input and returns some columns back.

In your case, your flowlet actually outputing back only one column which is RandSortCol. Hence, the moment you add that flowlet in to dataflow using flowlet transformation it is returning back only one column which is RandSortCol.

Kindly consider checking below video if mine, where i explained about flowlets.
Flowlets in Mapping data flow in Azure Data Factory

Hope this helps. Please let me know if any queries.

-----------

Please consider hitting Accept Answer button. Accepted answers help community as well.

Jan 1 Reputation point

2022-12-08T12:20:21.787+00:00

@ShaikMaheer-MSFT Thank you for your answer, I understand that at the moment it's intended behavior, but do you think in the future it would be possible to implement something like I intended with the flowlet? Since flowlets are intended as a reusable building block, it would be helpful to be able to use any input columns and not be restricted to certain input.
ShaikMaheer-MSFT 38,556 Reputation points Microsoft Employee Moderator

2022-12-16T05:25:42.863+00:00

Hi @Jan , Flowlets basic idea is to take some columns as input and give you back some columns are as output. In this process of taking input columns and generating output columns we can place some reusable transformation logic. So, It makes more sense when we use flowlet transformation it should give us back output columns only. Hope this helps.

------------

Please consider hitting Accept Answer button. Accepted answers help community as well.

Share via

Azure Data flow: Flowlets for any input

1 answer

Your answer