Azure data factory - ADF - Inline Sink Delta

Question

Azure data factory - ADF - Inline Sink Delta

jmarciogsousa 6

I'm getting the error for the ADF data flow - Delta Sink on key column, but I did the schema import and the column exists:
"Key column 'undefined' does not exist in mapped columns."

Some clue about the problem, looks like a bug, I know the Delta Sink is on preview

MarkKromer-MSFT 5,226 Reputation points Microsoft Employee Moderator

2020-08-16T22:26:21.42+00:00

Would you mind replying here with the data flow script behind this data flow? Copy and paste the script from the Script button in the data flow designer.
HarithaMaddi-MSFT 10,146 Reputation points

2020-08-18T06:39:37.827+00:00

Hi @jmarciogsousa ,

Are you still facing the issue? If your query is addressed, request you to mark the answer that helped you as “Mark as Answer” and "Up-Vote" so that other forum members can benefit from it. If you need any further help do let us know.
jmarciogsousa 6 Reputation points

2020-08-18T12:26:13.38+00:00

Hi @MarkKromer-8019 thank you for your feedback, I attached as requested18305-script.txt the script.
I changed the column to "HashId" and it's also on the mapped tab.

6 answers

Your answer

MarkKromer-MSFT 5,226 Reputation points Microsoft Employee Moderator

2020-08-16T22:26:21.42+00:00

Would you mind replying here with the data flow script behind this data flow? Copy and paste the script from the Script button in the data flow designer.
HarithaMaddi-MSFT 10,146 Reputation points

2020-08-18T06:39:37.827+00:00

Hi @jmarciogsousa ,

Are you still facing the issue? If your query is addressed, request you to mark the answer that helped you as “Mark as Answer” and "Up-Vote" so that other forum members can benefit from it. If you need any further help do let us know.
jmarciogsousa 6 Reputation points

2020-08-18T12:26:13.38+00:00

Hi @MarkKromer-8019 thank you for your feedback, I attached as requested18305-script.txt the script.
I changed the column to "HashId" and it's also on the mapped tab.

Answer 1

Jack Ma 161

Hi @jmarciogsousa @Wesley Tremayne ,

Thanks for reporting this issue! It’s a bug on our UX tool, it complains the key column as “undefined” because the schema of sink is not published so that the UX tool can’t recognize it properly. We will create a work item for it and fix this issue soon.

Meanwhile as a workaround, you can try:

Uncheck delete/upsert/update and only leave insert for the delta sink -> import the schema from “Mapping” panel then save/publish the data flow.
Once saved/published, click “Refresh” button on the page, navigate to the data flow and set delete/upsert/update + key columns again.

Wesley Tremayne 1 Reputation point

2020-10-21T13:26:26.423+00:00

@Jack Ma @Kiran-MSFT That worked! Thanks for the quick responses!

Answer 2

jmarciogsousa 6

I get it working with the procedure

Mapping
insert
publish
refresh page -> if not works go to 5.
turn off debug mode
delete and key column
publish

HarithaMaddi-MSFT 10,146 Reputation points

2020-08-19T06:11:43.31+00:00

Hi @jmarciogsousa ,

Glad that you found a resolution.

Please do consider to click on "Accept Answer" and "Up-vote" on your post, as it can be beneficial to other community members
Chris 1 Reputation point

2020-08-26T14:29:02.867+00:00
I'm seeing the same issue, but following the steps from @jmarciogsousa does not resolve the issue for me. I have tried the following:

Confirm that all key columns from the settings tab are mapped in the mapping tab

Deleted all key columns, un-ticked "Allow upsert", ticked "Allow insert", published, re-ticked "Allow upsert" and added only one of my key columns

I'm attaching the code in text format as well.
Kiran-MSFT 696 Reputation points Microsoft Employee

2020-08-27T05:12:28.9+00:00

Click on the script button which has the real details of the dataflow. It is the first button shaped like a scroll next to the code button.
Chris 1 Reputation point

2020-08-27T15:18:13.957+00:00

Here is the script
jmarciogsousa 6 Reputation points

2020-09-10T12:00:16.207+00:00

@Chris try to refresh the page after publish

Answer 3

Kiran-MSFT 696 Microsoft Employee

Check to make sure you have mapped "prrowid" column in the mapping tab. It is most likely left unmapped.

Answer 4

Hello Everyone,

I am facing the same issue. My requirement is to build metadata driven mapping data flow (All Parameterized way).

I have to read multiple CSV files (All are different files having differing schema) from raw_zone and do SCD 1 or 2 based on the parameterized key columns. I have enabled Azure-DevOps in my ADF. Working on feature branch and I could not publish from feature branch.

Key Points to Note:

In Source transformation, enabled schema drift and thus no projection.
In Sink transformation, specifying the key columns using byName('Drifted_Column') function. Auto-Mapping is selected in Sink.28170-dataflow-script.txt

Could you please help me for the solution ? @jmarciogsousa

Answer 5

jmarciogsousa 6

@Balan Murugan , you can try refresh the page if not stop-start debug mode

Balan Murugan 1 Reputation point

2020-09-25T09:03:54.883+00:00

@jmarciogsousa Tried it multiple times. But still, no luck. When adding another sink as Azure SQL DB it does not throw this error. It is erroring out ONLY when the sink is in DELTA forma.
Balan Murugan 1 Reputation point

2020-09-25T09:05:52.403+00:00

In the attached screen shot, sink1 is for delta format. sink2 is for Azure SQL DB.

I believe, since delta format is in public preview, this particular feature is not yet available I guess for the parameterization.

Share via

Azure data factory - ADF - Inline Sink Delta

6 answers

Your answer