How to access files in nested folders

smruti ranjan 20 Reputation points
2023-12-05T18:28:12.99+00:00

I have a container named "bronze" with the following folder structure:

production/product/product.csv
production/productcategory/category.csv

Using the metadata activity, I am only able to retrieve the metadata of bronze container using childitems and get the first folderproduction . How can I get the metadata of the folders inside production (i.e., product and productcategory) and then retrieve the files inside those folders?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,153 questions
0 comments No comments
{count} votes

Accepted answer
  1. phemanth 8,645 Reputation points Microsoft Vendor
    2023-12-06T10:52:32.5966667+00:00

    @smruti ranjan

    Thanks For Reaching out MS Q&A

    I have repro’d by iterating through multiple sub-folders inside For Each activity using execute pipeline activity.

    Source dataset:

    Create a dataset for the source and add the dataset parameter for passing the value dynamically.

    enter image description here

    Main pipeline:

    1. Using the Get Metadata activity, get the folders inside the given container.

    enter image description here

    enter image description here

    1. Pass the child items to the ForEach activity. Inside ForEach, add execute pipeline to call another pipeline to get the subfolder for each current item (@item().name).

    enter image description here

    Child pipeline1 (to get the subfolders):

    1. In the child pipeline, create a pipeline parameter to get the current item name (main folder name) from the parent pipeline.

    enter image description here

    1. Using the Get Metadata activity, get the subfolders list. Use the parameters in the dataset.

    Dataset property value: @concat(pipeline().parameters.dir1,'/')enter image description here

    enter image description here

    1. Pass the child items to ForEach activity. Inside ForEach, you use filter activity to filter out the sub folder name if required. Then pass the required current item to execute pipeline activity to call the child pipeline (which gets the files from each sub folder). Pass the child pipeline parameter value from here.

    @concat(pipeline().parameters.dir1,'/',item().name,'/')enter image description here

    Child pipeline2 (gets the files and process):

    1. Create the pipeline parameter to get the value from its parent pipeline.

    enter image description here

    1. Using the Get Metadata activity get the files from each sub folder by passing the parameter value to the dataset parameter.

    enter image description here

    enter image description here

    Pass the output child items to ForEach activity. Inside ForEach, you can use filter activity to filter out the files.

    Using Copy data activity to can copy the required files to the sink.

    Dataset properties:

    Dir - @concat(pipeline().parameters.path,'/',item().name)enter image description here

    Hope this helps. Do let us know if you any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful