Catch exception (if any) of a Copy activity

Question

I have 2 simple copy activities.

FromSourceToStaging --(Success)-->CatchException.

First One (named as FromSourceToStaging) copies a csv file into a table. This is set to 'Skip incompatible rows' to a specific folder. Upon success, the second one (named as CatchException) reads the exceptions generated by the previous activity (if any) and writes it to a table.

CatchException activity's Source path is set as a dynamic content @activity('FromSourceToStaging').output.logFilePath, this path will only exist if there were any rows skipped by the FromSourceToStaging. It wont exist if there were no rows skipped.

Often when there were no errors in FromSourceToStaging, the next activity CatchException complaints that path not found (e.g myfolder/copyactivity-logs/FromSourceToStaging/{guid})

The funny part is that it is working without any issues in another environment.

What's the best approach to handle this?

Answer 1

@HimanshuSinha-msft

The logic is a simple, default behaviour present in ADF.

Copy activity is set to 'Skip incompatible rows'. When set, ADF creates a directory named as Activity RUNID in the destination whenever it skips some rows. If no rows are skipped then this directory doesn't exist.

The other pipeline has identical data where ADF happily works without a complaint. I am not sure if there has been any changes to ADF functionality.

Source to Staging:

Catch Exceptions (if any) :

Failure:

Error:
FileSystem: 'migration-exceptions'. Path: 'migration-exceptions/copyactivity-logs/SourceToStaging/607eb703-5250-4f2f-8012-cf43538b8eb0'. ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'. RequestId: '9ef525c5-e01f-0005-0a57-5c70fb000000'. TimeStamp: 'Tue, 08 Jun 2021 11:15:00 GMT'.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Azure.Storage.Data.Models.ErrorSchemaException,Message=Operation returned an invalid status code 'NotFound',Source=Microsoft.DataTransfer.ClientLibrary,'