Azure Data Factory Sink issue - Sink flow is changing column data

Daniel Widenhouse 0 Reputation points
2023-10-25T19:12:21.4533333+00:00

In Azure Data Factory, I have a data flow which takes data from a simple SQL table, adds some derived columns and a surrogate key, and then pushes into another SQL table. All data previews and CSV exports work correctly up until the sink component. When viewing the data in the data preview on the sink, it is perfect.

User's image

When exporting the list to CSV, it is not. When pushing the data through to the new table, I get an error because the unique identifier is duplicated. All of the columns have shifted and surrogate key is lost.

User's image

The surrogate key is included in the mapping.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,653 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Harishga 6,005 Reputation points Microsoft External Staff
    2023-10-26T11:07:00.2333333+00:00

    Hi@Daniel Widenhouse ,

    Welcome to Microsoft Q&A platform and thanks for posting your question here.

    As I understand that there seems to be an issue with the Azure Data Factory Sink component.

    Here are some possible reasons for the issue:

    Check the mapping of the surrogate key column in the sink component. Ensure that the surrogate key column is mapped correctly in the sink component.

    Check the data types of the columns in the source and sink tables. Ensure that the data types of the columns in the source and sink tables are the same. If the data types are different, it can cause issues with the data flow.

    Check the data flow settings. Ensure that the data flow settings are configured correctly.

    while doing repro from my end I couldn't see any issue. please find the screenshots.

    20231

    2023

    20232

    I hope this information helps you. Let me know if you have any further queries.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.