Issue with Conditional Split in ADF Data Flow Creating Empty Sink File

Garth Gabriels 0 Reputation points
2024-01-26T07:33:28.3666667+00:00

In my Azure Data Factory Data Flow, I'm encountering an issue with a Conditional Split transformation that's leading to the creation of an empty file in the sink. My Data Flow includes a Conditional Split with two conditions:

  1. !isNull(Field)
  2. isNull(Field)

The intention is to direct rows based on whether **Field**is null or not. Here's the issue I'm facing:

  • Despite properly routing rows based on these conditions, the sink connected to the isNull(Field) condition is creating an empty file even when there are no rows that meet this condition.
  • I expected that if no rows satisfy isNull(Field), no file would be created in the associated sink. However, an empty file is being generated regardless.
  • This behavior occurs even when the dataset does not contain any records where **Field**is null.

I'm looking for insights or solutions to prevent the creation of this empty file in cases where the Conditional Split results in no rows meeting the isNull(Field) condition. Attempted Solutions and Observations:

  • The Conditional Split itself seems to function correctly, as it properly routes rows based on the conditions.
  • The issue appears to be with the Sink operation in the Data Flow, which executes and creates a file even when the incoming stream is empty.
  1. Is there a way within ADF Data Flow to conditionally prevent a sink from creating a file if the incoming stream from a Conditional Split is empty?
  2. Are there any recommended practices or workarounds in ADF to handle this scenario more effectively?

Any advice or guidance on this issue would be greatly appreciated.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,219 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Amira Bedhiafi 28,381 Reputation points
    2024-01-26T10:26:33.9133333+00:00

    In ADF Data Flows, the sink settings can influence file generation. There's a possibility that the sink is configured in a way that leads to the creation of an empty file regardless of whether the incoming stream contains data. Before the data reaches the sink, you could add an additional transformation that checks the row count. If the row count is zero, you can divert the flow to avoid writing to the sink. This might involve adding a derived column or an aggregate transformation to count rows and then using another Conditional Split based on this count. If the standard transformations don't provide a solution, consider using a script-based transformation (if supported in your ADF version) to implement custom logic for file creation.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.