How to access files in nested folders

Question

I have a container named "bronze" with the following folder structure:

production/product/product.csv
production/productcategory/category.csv

Using the metadata activity, I am only able to retrieve the metadata of bronze container using childitems and get the first folderproduction . How can I get the metadata of the folders inside production (i.e., product and productcategory) and then retrieve the files inside those folders?

Answer 1

@smruti ranjan

Thanks For Reaching out MS Q&A

I have repro’d by iterating through multiple sub-folders inside For Each activity using execute pipeline activity.

Source dataset:

Create a dataset for the source and add the dataset parameter for passing the value dynamically.

Main pipeline:

1. Using the Get Metadata activity, get the folders inside the given container.

1. Pass the child items to the ForEach activity. Inside ForEach, add execute pipeline to call another pipeline to get the subfolder for each current item (@item().name).

Child pipeline1 (to get the subfolders):

1. In the child pipeline, create a pipeline parameter to get the current item name (main folder name) from the parent pipeline.

1. Using the Get Metadata activity, get the subfolders list. Use the parameters in the dataset.

Dataset property value: @concat(pipeline().parameters.dir1,'/')

1. Pass the child items to ForEach activity. Inside ForEach, you use filter activity to filter out the sub folder name if required. Then pass the required current item to execute pipeline activity to call the child pipeline (which gets the files from each sub folder). Pass the child pipeline parameter value from here.

@concat(pipeline().parameters.dir1,'/',item().name,'/')

Child pipeline2 (gets the files and process):

1. Create the pipeline parameter to get the value from its parent pipeline.

1. Using the Get Metadata activity get the files from each sub folder by passing the parameter value to the dataset parameter.

Pass the output child items to ForEach activity. Inside ForEach, you can use filter activity to filter out the files.

Using Copy data activity to can copy the required files to the sink.

Dataset properties:

Dir - @concat(pipeline().parameters.path,'/',item().name)

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.