@Venkatesh Srinivasan - Thanks for the question and using MS Q&A.
You can use a Validation in a pipeline to ensure the pipeline only continues execution once it validates the attached dataset reference exists, that it meets the specified criteria, or times out.
To check if the file present inside the subfolder before proceeding, please follow the below steps:
- Create a storage account named "storageaccount5086" and set up two containers: input and output. In the input container, I created a folder named "json" and uploaded three JSON files into it.
Create and open a Synapse workspace, then create a pipeline named "filevalidation." Search for the validation activity and drag and drop it into the pipeline.
Click on Settings and add a dataset named "sourcedataset1." Browse to select the folders without selecting any files, then click OK.
In the settings, set the timeout to 30 seconds, select TRUE to check if the folder exists and contains items, then click on Validate and Debug.
It take some time, you will see that the pipeline status shows as succeeded.
Click on Output, which will display "True" and list the three files we uploaded, as shown in the figure below.
- Create a Get Metadata activity and name it "Get Metadata1." Connect the output of the Validation activity to the input of the Get Metadata activity. In the settings, select the dataset "sourcedataset1" that we previously created and choose "Child Items" in the field list. Then click on Validate and Debug. After a moment, you will see that the pipeline executed successfully.
- As a result, you will see the child items in the output, as illustrated in the figure below.
Now create a ForEach activity and connect the output of the Get Metadata activity to the input of the ForEach activity. In the settings, select "Sequential" and click on "Add dynamic content." In the activity outputs, select "Get Metadata1 Childitems," then click OK.
- Click on Validate and Debug, and you will see that the pipeline has succeeded.
Click on Output, and you will see the files that are present in the folder.
For more details, refer to Validation activity in Azure Data Factory and Synapse Analytics pipelines
Hope this helps. Do let us know if you have any further queries.
------------
If this answers your query, do click `Accept Answer`
and `Yes`
for was this answer helpful. And, if you have any further query do let us know.