Azure synapse analytics data flow conditional split is sending empty file with headers to non-matching condition

Question

In the data flow I have an assert transformation that is asserting columns exist (using hasColumns('ColumnName', 'StreamName)) in the csv source then a conditional split using isError == false() for if there are missing columns all rows will error

I'm encountering issue where if all columns exist it will go to the correct conditional split headers with data and all but also go to the other split and the sink creates a file but with just headers (no data)

How do I prevent empty file with just headers from being created since there are 0 records?

Answer 1

Hello @Alpine GreenT
Just checking if we are still connected on this discussion? Please let us know if you need to add more info so that we better assist you!

In addition to the above solution posted by Subashri Vasudevan, please also consider below suggestion!

You can add another condition to the conditional split transformation to check if the row count is greater than 0. This will ensure that only rows with data are sent to the sink transformation.

Answer 2

Hi , if you don't want to use the other flow in the conditional split that creates an empty file, you could probably delete that flow. Please post your data flow screenshot for more clarity. Thanks