How to check if the file present inside the subfolder before proceeding?

Venkatesh Srinivasan 0 Reputation points
2024-10-23T11:52:29.1233333+00:00

Any possible way I can check subfolders contains file or not? Get meta data activity doesn't seems working well with the subfolders

I just want to check if file exist in any of the subfolder, Looks simple but I can't find any solution for this, Much appreciate if someone could throw light on this

Folder path :
inventory / raw / A
inventory / raw / B
inventory / raw / C

User's image

User's image

I passed the child item output result to the for each activity

User's image Struggling to read the subfolder files from another get meta activity inside the foreach loop now.
User's image

Highly appreciate if you have any other work around, thanks

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,997 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Ganesh Gurram 1,110 Reputation points Microsoft Vendor
    2024-10-24T08:28:46.9166667+00:00

    @Venkatesh Srinivasan - Thanks for the question and using MS Q&A.

    You can use a Validation in a pipeline to ensure the pipeline only continues execution once it validates the attached dataset reference exists, that it meets the specified criteria, or times out.

    To check if the file present inside the subfolder before proceeding, please follow the below steps: 

    • Create a storage account named "storageaccount5086" and set up two containers: input and output. In the input container, I created a folder named "json" and uploaded three JSON files into it. 

    User's image

    Create and open a Synapse workspace, then create a pipeline named "filevalidation." Search for the validation activity and drag and drop it into the pipeline.

    User's image

     

    Click on Settings and add a dataset named "sourcedataset1." Browse to select the folders without selecting any files, then click OK. 

    User's image

    In the settings, set the timeout to 30 seconds, select TRUE to check if the folder exists and contains items, then click on Validate and Debug. 

    User's image

    It take some time, you will see that the pipeline status shows as succeeded. 

    User's image

    Click on Output, which will display "True" and list the three files we uploaded, as shown in the figure below.

    User's image

     

    • Create a Get Metadata activity and name it "Get Metadata1." Connect the output of the Validation activity to the input of the Get Metadata activity. In the settings, select the dataset "sourcedataset1" that we previously created and choose "Child Items" in the field list. Then click on Validate and Debug. After a moment, you will see that the pipeline executed successfully.   

    User's image

    • As a result, you will see the child items in the output, as illustrated in the figure below. 

    User's image

    Now create a ForEach activity and connect the output of the Get Metadata activity to the input of the ForEach activity. In the settings, select "Sequential" and click on "Add dynamic content." In the activity outputs, select "Get Metadata1 Childitems," then click OK. 

    User's image

    • Click on Validate and Debug, and you will see that the pipeline has succeeded. 

    User's image

    Click on Output, and you will see the files that are present in the folder. 

    User's image

    For more details, refer to Validation activity in Azure Data Factory and Synapse Analytics pipelines

    Hope this helps. Do let us know if you have any further queries.

    ------------  

    If this answers your query, do click `Accept Answer` and `Yes` for was this answer helpful. And, if you have any further query do let us know.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.