Possible bug with flowlets output mapping

Question

Possible bug with flowlets output mapping

John Neubecker 86

I've run into a possible bug with flowlet outputs, unless I'm doing something wrong.

The flowlet output

Setting the mapping in a data flow

The dropdown shows the schema from another one of the flowlets output streams

John Neubecker 86 Reputation points

2022-07-01T15:38:25.387+00:00

It looks like this bug also affects using output columns in other transformations even when using byName with the flowlet stream name.

Spark job failed: {
Higher order function cannot be evaluated with prior data types\nDF-EXPR-020 at Source 'ExistingEntitiesResult'(Line 31/Col 90): Higher order function cannot be evaluated with prior data types\nDF-EXPR-010 at Output 'ExitingEntities'(Line 60/Col 10): Column 'Lease_ID' not found. The stream is either not connected or column is unavailable\nDF-EXPR-010 at Output 'ExitingEntities'(Line 61/Col 10): Column 'DB_DocumentCollectionId' not found. The stream is either not connected or column is unavailable\nDF-EXPR-010 at Output 'ExitingEntities'(Line 62/Col 10): Column 'DB_ExtractSetId' not found. The stream is either not connected or column is unavailable\nDF-EXP-006 at Source 'Attribut\"}}\n"
} - RunId: 5f50bfe4-e6cb-4d1b-918d-3649a7eb5b0f

Lease_ID isn't even part of the stream, there's a parameter to the flowlet that takes a key column name and uses it in the select list for the flowlet and Lease_ID is the default value but a different value is being passed in this case.

Accepted answer

1 additional answer

Your answer

John Neubecker 86 Reputation points

2022-07-01T15:38:25.387+00:00

It looks like this bug also affects using output columns in other transformations even when using byName with the flowlet stream name.

Spark job failed: {
Higher order function cannot be evaluated with prior data types\nDF-EXPR-020 at Source 'ExistingEntitiesResult'(Line 31/Col 90): Higher order function cannot be evaluated with prior data types\nDF-EXPR-010 at Output 'ExitingEntities'(Line 60/Col 10): Column 'Lease_ID' not found. The stream is either not connected or column is unavailable\nDF-EXPR-010 at Output 'ExitingEntities'(Line 61/Col 10): Column 'DB_DocumentCollectionId' not found. The stream is either not connected or column is unavailable\nDF-EXPR-010 at Output 'ExitingEntities'(Line 62/Col 10): Column 'DB_ExtractSetId' not found. The stream is either not connected or column is unavailable\nDF-EXP-006 at Source 'Attribut\"}}\n"
} - RunId: 5f50bfe4-e6cb-4d1b-918d-3649a7eb5b0f

Lease_ID isn't even part of the stream, there's a parameter to the flowlet that takes a key column name and uses it in the select list for the flowlet and Lease_ID is the default value but a different value is being passed in this case.

Answer 1

ShaikMaheer-MSFT 38,546 Microsoft Employee Moderator

Hi @John Neubecker ,

Thanks for raising this bug and share details on it. After discussions and internal repro of scenarios PG acknowledged it as bug and then took it on priority to deploy fix for this issue.

Now, we can see multiple output columns of flowlet in flowlet transformation mappings. Kindly check and your end too.

Hope this helps. Please let us know if any further concerns. Happy to take them to PG team.

As Bug fixed now, If all good I would encourage you to retake survey. Looking forward for your comments. Thank you.

--------------

Please consider hitting Accept Answer and Up-vote button. Accepted answers help community as well.

ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-08-01T10:03:00.733+00:00

Hi @John Neubecker ,

Just checking in to see if the above answer helped. If this answers your query, do click and upvote for the same. And, if you have any further query do let us know.

Answer 2

ShaikMaheer-MSFT 38,546 Microsoft Employee Moderator

Hi @John Neubecker ,

Thanks for posting question in Microsoft Q&A Platform.

I tried to reproduce scenario by creating two flowlets. One has output columns personId and personName. Another has output columns empId & empName. When I used these flowlets in Flowlet transformation, I am seeing correct column only in Mapping tab of flowlet.

Please check below screenshots.

I am using Source file with input columns id,name,gender & doj. and when I used flow transformation with personFlowlet I am using dropdown with Source transformation columns left side and right side flowlet output columns correctly.

So in your case, its showing source transformation related columns in the dropdown of leftside inside mapping tab of flowlet transformation.

Kindly check below video to understand more about flowlets.
Flowlets in Mapping data flow in Azure Data Factory

Hope this helps. Please let us know if any further queries.

--------
Please consider hitting Accept Answer button. Accepted answers help community as well.

John Neubecker 86 Reputation points

2022-07-05T13:32:08.71+00:00

This isn't the same as my scenario. Both my outputs are part of the same flowlet, specifically it's a single flowlet with 3 sources and 3 outputs, and all three need mapped in the data flow directly to another flowlet. I was able to recreate this issue again just now with a new dataflow. Having one flowlet with 2 outputs and another flowlet with 2 inputs and trying to map the outputs from one to the inputs of the other. I suppose I could break it up into 3 different flowlets as a workaround but that limits some of the utility gained by using flowlets to group things that will always go together. It seems to be a bug with flowlets that have multiple outputs. The video you linked doesn't have any examples of a flowlet with multiple outputs. Thinking about this more I can probably work around the issue by having a derive column transformation between the flowlets but it would still be good if this could be done without adding a transformation in between.
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-07-07T08:56:50.77+00:00

Hi @John Neubecker ,

Thanks for sharing more details with example scenario. This helps to understand the issue better. You are absolutely right, seems like bug to me.

There two possible bugs I observed.

Bug1#: When an flowlet has multiple outputs. In dataflow, flowlet trasformation, mapping tab we could see only first output values.

Bug2#: When we use two flowlets transformations one after another then, we could see in second flowlet transformation under mapping tab input columns are not coming up from first flowlet.

@MarkKromer-MSFT Kindly check this once and share your thoughts. Thank you.
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-07-20T05:00:47.207+00:00

Hi @John Neubecker ,

We had discussion with Product team about this bug and its confirmed there end too as bug. Created a BUG item to resolve this internally. I will be updating thread once I get updates on resolution. Thank you.

Share via

Possible bug with flowlets output mapping

1 additional answer

Your answer