Share via

Some Rows are getting dropped while sinking in CSV format in Azure Data Factory using Data flows

Shivam Kumar 0 Reputation points
2024-07-09T06:18:37.7666667+00:00

I am using csv as a row, and tranforming it in Azure Data flow in Pipeline.

while sinking it in CSV format, some rows are getting dropped, but they are present in data preview when I am adding filter condition to look that particulat ID.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,042 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Amira Bedhiafi 18,741 Reputation points
    2024-07-09T08:30:48.2533333+00:00

    The possible issue for the behaviour you are having

    1. Data Type Mismatches
    2. Null or Special Characters
    3. Filter Conditions
    4. Schema Mismatch

    If you find that null values are causing rows to drop, you might want to replace nulls with a default value before sinking:

    
Derived Column Transformation:

ReplaceNull(column_name, default_value)

    If the issue persists after these checks, you might want to provide more specific details or screenshots of your data flow configurations to further diagnose the problem.

    0 comments