Data flow sink bug, mapped column not being written to

John Neubecker 81

I have a data flow where in the transformation before a sink a column exists and has a value, key=86.

It's mapped in the sink

But it does not even appear as a column in the sink preview nor does it get inserted if I run the data flow in a pipeline. I'm not showing the preview where the column isn't there because it has other data I don't want to post here. I've tried deleting and readding the sink and that doesn't do anything. I know you'll probably try this in a simple dataflow and it will work. I've never seen this issue before. It's probably some edge case that hasn't been tested. Probably something to do with the JSON generated when you have a factory backed by git.

John Neubecker 81 Reputation points

2022-09-29T20:41:19.78+00:00

Actually it looks like this is related to a recurring bug with using flowlets where the output columns need to be explicitly rederived. This is the third time I've seen this bug happen and after a couple days it goes away, probably because of another patch release that fixes it until another release breaks it again. Please tell the product team who maintains ADF to add better testing around flowlets. It's frustrating, time consuming, and money wasting to debug these sorts of issues. The workaround for this is to have a derive transformation after using a flowlet and use the byName function to explicitly get the output columns of the flowlet. It's very strange because these columns show up like they're drifted columns after the flowlet is run but they can't be in a sink.
John Neubecker 81 Reputation points

2022-09-29T20:43:45.84+00:00

It's possible this only happens when multiple flowlets are used.
John Neubecker 81 Reputation points

2022-09-29T20:48:06.097+00:00

Correction. The workaround is to add a select transformation right after running the flowlet to select all the output columns. It makes no sense that this makes any difference.
MithunKumarRajendran-MSFT 1 Reputation point

2022-09-30T17:44:42.537+00:00
Hi. Looking to gather some more details on the issue.

Is the column "Key" an unmapped column at the flowlet?

Is the issue happening currently?

It would be great if you could share the most recent dataflow run id for the run with this issue for us to look into this further.
John Neubecker 81 Reputation points

2022-09-30T18:39:10.95+00:00

The key column is mapped. It is happening if I remove the work-around. I've run a lot of pipelines and don't remember which ones this happened with. Is there a way to see what data was inserted by a pipeline run?
BhargavaGunnam-MSFT 26,496 Reputation points Microsoft Employee

2022-10-10T23:20:02.72+00:00

Thank you @MithunKumarRajendran-MSFT

Hello @John Neubecker ,
I am just checking in to see if you need any further assistance here.

In case you need further assistance, it would be great if you could share the dataflow run id as requested by @MithunKumarRajendran-MSFT