ADF skip and log incompatible rows - always creates empty file

Question

ADF skip and log incompatible rows - always creates empty file

O'Flynn, Kieran 36

We have a number of ADF pipelines with copy activities that load raw data files into Azure SQL Paas DB's or Azure DWH staging tables.

We have "skip incompatible rows" fault tolerance switched on, and we are logging the skipped rows to blob storage rejection files. Logging level is "Warning" and logging mode is "Reliable".

We have found that this always creates a reject file in the blob storage container. The file will be empty except for the ADF column headers (see image).

Is there any way to avoid creating these empty files while still having this logging capability?

Accepted answer

0 additional answers

Your answer

Answer 1

AnnuKumari-MSFT 34,556 Microsoft Employee Moderator

Hi @O'Flynn, Kieran ,
Welcome to Microsoft Q&A platform and thankyou for posting your query.
As I understand your question here, you want to know if there is any way to avoid the creation of empty files generated as a result of no warnings for skip incompatible rows option.

Unfortunately, even if there is no warning to be captured the files would be generated . That's how it works right now.

However, let me collaborate with the internal product team to find out if there is any future plan to just avoid the creation of files if there is no warning . I will keep you posted once I hear back from them . Thanks for your patience!

O'Flynn, Kieran 36 Reputation points

2022-05-03T08:38:46.013+00:00

Thank you AnnuKumari.

It would be great if this could be added to the product.

At the moment we have 1000's of these empty files and we are having to code a solution to look at the reject file that has just been created and if less than 100kb delete it, for each file we load. That is a lot of extra activities executing each day/month/year.
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2022-05-09T11:14:19.757+00:00

Hi @O'Flynn, Kieran ,
Appreciate if you could share the feedback on our Azure data factory feedback channel, which would be open for the user community to upvote & comment on. This allows our product team to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

If the suggested response helped you, please click Accept answer as accepted answer helps the community as well.
O'Flynn, Kieran 36 Reputation points

2022-05-10T14:58:30.83+00:00

Done now.

Thanks

Share via

ADF skip and log incompatible rows - always creates empty file

0 additional answers

Your answer