BUG - Automatic remapping in Dataflow for Flowlet in ADF

Question

BUG - Automatic remapping in Dataflow for Flowlet in ADF

Gużewski, Jacek 135

I have a problem with incorrect working UI in ADF. I think, that is not only problem with UI because when I am running this dataflow it's ended up with failed status.

Problem is:

I have created a few dataflows with flowlets within. I have used this many times and their work perfectly. Few day ago I found problem with one and tried to figure out where is the problem and I found this.

Dataflow looks like this:
User's image

and code is like this, where marked red is part for map attributes for created flowlet:
User's image

If I go to the Flowlet to the mapping settings, suddenly appears this:
User's image

System now showing me my mapping (like you can see in code), but mapping first attribute from previous stage to both columns, without any information that dataflow has changed (no request for publishing).
I have the same in all my dataflows where I've used flowlets. Sth it looks like this, where I've more than one version of mapping in code :
User's image

If someone has had the same and found where a problem is and/or found how to correct this, I would be grateful for the information.

Thank in advance.

Jack.

Anonymous

2023-09-25T10:16:57.86+00:00

@Gużewski, Jacek

Thanks For approaching Microsoft Q&A

can you please do some checks given below and confirm us?

1 Check the mapping settings for the flowlet in question to ensure that they are correct.

2 Check if there are any changes in the source data that might have caused the issue.

3 Recreate the flowlet and see if that resolves the issue.

4 Create a new dataflow and copy the flowlet to the new dataflow to see if that resolves the issue.

5 Check if there are any updates or patches available for Azure Data Factory that might address the issue.

for further queries please do let us know.
Gużewski, Jacek 135 Reputation points

2023-09-25T10:25:40.3133333+00:00

Ad1. All mapping for the flowlet were correct till the change was made out of my control.

Ad2. There are no changes in the source datasets.
Ad3. Done, but I have the same issue in the brand new dataflow

Ad4. Still doesn't work correctly.
Ad5. What do you mean updates or patches for ADF? It is SaaS I do not menage it.
Anonymous

2023-09-25T12:58:44.8933333+00:00

I am also facing the same issue. Even if i change the vaule and publish, it gets removed after referesh
Rafael Soria Diez 0 Reputation points

2023-09-25T13:01:28.9466667+00:00

I am also having the same problem. I have tried the above solutions and nothing works
OLLIVO Francoise 5 Reputation points

2023-09-26T12:10:29.2866667+00:00

We have the same issue here. In our case, the workarround is to rename the input columns (their name must be the same than the flowlet output columns). Then update the .JSON to keep only one map (the one without the "=")
Gużewski, Jacek 135 Reputation points

2023-09-26T12:20:32.74+00:00

Thanks OLLIVO,
I found this workaround, but it is not suitable for me because I use flowlets for validation attributes in datasets form csv files (like emptiness, format suitable for converting, limited list of values, etc) and I have more than one flowlet in one dataflow and I would have to add select stage, before each flowlet, only for changing name similar to input name in flowlet - too many changes.
Anonymous

2023-09-27T07:21:42.73+00:00

Hi@Gużewski, Jacek we appreciate your patience, to resolve this we are working with our internal team, and we get back to you shortly.
Sakaar Mishra 11 Reputation points

2023-09-27T14:49:05.77+00:00

We are facing similar issue, where flowlet mapping for all the columns getting automatically updated with the first column post refresh, which is ending up having compilation errors on already published code.

We first encountered this issue on Friday, 22nd Sept, has there been any patch released in background causing this issue ?
Also my subscription is of UK south so would be interested to know if other regions are also impacted.
Ha_Ka 0 Reputation points

2023-09-27T14:58:24.45+00:00

I have exactly the same issues.

Additionally when having multiple flowlets with only one mapping column a comma gets added in the script which makes all pipeline runs fail:

Error msg from pipeline run:

Operation on target Data flow1 failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: com.microsoft.dataflow.broker.InvalidOperationException: DSL compilation failed: DF-DSL-001 - DSL stream has parsing errors\nLine 454 Position 5: extraneous input ',' expecting {DECIMAL_LITERAL, HEX_LITERAL, OCT_LITERAL, BINARY_LITERAL, MAX_INT, MIN_INT, MAX_LONG, MIN_LONG, POSITIVE_INF, NEGATIVE_INF, '-', '!', '$', '~', ':', '(', '#', '[', '@(', '[]', FLOAT_LITERAL, HEX_FLOAT_LITERAL, STRING_LITERAL, REGEX_LITERAL, 'parameters', 'functions', 'stores', 'as', 'input', 'output', 'constant', 'expression', 'integer', 'short', 'long', 'double', 'float', 'decimal', 'boolean', 'timestamp', 'date', 'byte', 'binary', 'integral', 'number', 'fractional', 'any', IDENTIFIER, ANY_IDENTIFIER, META_MATCH, '$$', '$$$', '$#', OPEN_INTERPOLATE}\nLine 465 Position 5: extraneous input ',' expecting {DECIMAL_LITERAL, HEX_LITERAL, OCT_LITERAL, BINARY_LITERAL, MAX_INT, MIN_INT, MAX_LONG, MIN_LONG, POSITIVE_INF, NEGATIVE_INF, '-', '!', '$', '~', ':', '(', '#', '[', '@(', '[]', FLOAT_LITERAL, HEX_FLOAT_LITERAL","Details":""}
Anonymous

2023-10-04T07:49:38.6533333+00:00

@Gużewski, Jacek

Thanks for flagging this. It looks we need a deeper look on this. Kindly consider having a support case so that engineering team can work with you and look the issue and help with it

for further queries pls do let us know
Anonymous

2023-10-04T08:17:07.74+00:00

Bug is fixed by the Azure team last week.
Gużewski, Jacek 135 Reputation points

2023-10-04T08:36:51.99+00:00

@Anonymous I have raised a ticket in Microsoft *Support request 2309250050000385), but I still haven't received any information that the issue was resolved, or any estimation of when it will be possible, or any official statement that this is a bug.
Anonymous

2023-10-04T08:40:28.77+00:00

@Gużewski, Jacek Can you try replicating the issue. because I raised it starting of last week and in the middle of the week it was fixed and I no longer face this issue.
Gużewski, Jacek 135 Reputation points

2023-10-04T08:40:56.63+00:00

@Anonymous I haven't received any information from MS in my support request about it. I need an official statement to start work with the tool - this is a commercial solution and I can't take responsibility if this part not working properly.
Gużewski, Jacek 135 Reputation points

2023-10-04T08:51:41.9566667+00:00

@Jayaraman, SS (Sabarish Subbramaniam) (eu) I checked I don't see problem - looks OK. But when I set the ticket for this issue I need an official confirmation if it is fixed to use this tool in our solution.
Anonymous

2023-10-05T13:21:34.5933333+00:00

My company has premium support, maybe that is why we got quick update (not sure).