How to avoid copy activity to error out , if there are zero files on the source end

Question

Hello team,

I am trying to copy data from Amazon S3 to Azure SQL server using copy activity. When there are no files in the S3 bucket, the copy activity gives an error that the bucket doesn't exist. I tried to use Lookup activity, but I wasn't able to get the count of rows from Lookup activity. Also I saw that there is limitation to number rows Lookup activity can read . Also, i dont want to use Dataflow.

Could you please suggest a way to if there are no files in S3 the copy activity should not be executed, instead of throwing an error.

Thankyou!

Answer 1

Hi @Shweta Chavan
To check if there are any files in a folder using the Get Metadata activity in Azure Data Factory, you should use the childItems property. This property will give you a list of files and subfolders inside the specified folder. If you want to check for the existence of files or folders, you can use the exists property, which will return true if the file or folder exists, and false otherwise.

Here’s how you can fetch the output to know if files exist or not:

@activity('YourGetMetadataActivityName').output.childItems

And to check for the existence of a specific file or folder:

@activity('YourGetMetadataActivityName').output.exists

If you need to check for multiple subfolders and their files, you need to set up additional Get Metadata activities or use a For Each loop to iterate through the child Items list.

Remember, the exists property will only be available in the output if you have specified it in the field list of the Get Metadata activity.

Reference
https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/data-factory/control-flow-get-metadata-activity.md

https://learn.microsoft.com/en-us/azure/data-factory/control-flow-get-metadata-activity

I hope this information helps you. Let me know if you have any further questions or concerns.