Azure Data Flow Not reading Entire Csv File

Rajput, Ravi 5

Hi ,
I have azure data flow that reads the csv file with 15M rows.
This data flow does lot of transformation like joining, select, filter etc.
it was working fine until last month, however now it is not reading the entire file. Out of 15M rows it is reading only 10M rows from .csv file.
I checked in the monitor, only 10M rows were read by source activity. Original Data Flow run ID : 9f62d4db-762c-4137-b321-e837ae562f10

I created a test data flow by removing all the other activities except the source and sink and tried running it, this time the data flow reading the entire file.

Test Data Flow run Id : 0b6d3743-949f-450f-842f-d29206b0cc41

phemanth 8,645 Reputation points Microsoft Vendor

2024-01-19T09:04:55.0533333+00:00
@Rajput, Ravi

Thanks for reaching MS Q&A

It seems like the issue is not with the source or sink activities, as your test data flow is able to read the entire file. The problem might be with one of the transformation activities in your original data flow.

Here are a few things you can check:

Data Skewness: Check if there’s a skew in data that’s causing an uneven distribution of data across partitions. This might cause some partitions to process more data than others, leading to timeouts or failures.

Transformation Logic: Review the transformation logic in your data flow. There might be conditions or filters that are inadvertently filtering out rows.

Resources: Ensure that your Azure Integration Runtime has sufficient resources to handle the volume of data. You might need to scale up your resources.

Debugging: Enable verbose logging for your data flow and check the logs for any errors or warnings that might indicate what’s going wrong. Hope this helps. Do let us know if you any further queries.
phemanth 8,645 Reputation points Microsoft Vendor

2024-01-22T11:13:29.9+00:00

@Rajput, Ravi We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
phemanth 8,645 Reputation points Microsoft Vendor

2024-01-23T10:20:13.2166667+00:00

@Rajput, Ravi just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Rajput, Ravi 5 Reputation points

2024-01-23T10:25:18.0366667+00:00

Hi, Not resolved yet, but I am checking with data/values if something is dropping.
phemanth 8,645 Reputation points Microsoft Vendor

2024-01-24T10:26:06.9566667+00:00

@Rajput, Ravi
kindly check the data and values to see if anything is dropping. If you find issues, please confirm us.

Share via

Azure Data Flow Not reading Entire Csv File