Accessing file name from nested folder structure and passing it to Notebook activity as a parameter in ADF.

Vinay5 46

Hello,

I have around 6 files in a nested folder structure as below.

data/year/Month/Day/time
File1.xml
File2.xml
... File6.xml.

I have to pass each file as a parameter to 6 different notebook activities.

So, I started to pass File1.xml as a parameter to a notebook activity. I used getmetadata activity and in field list I selected child items. It is giving only the year folder as output. I am unable to access file name from the nested folder.

Could you please let me know how to achieve this output.

Thank you.

prk 1 Reputation point

2022-10-19T19:50:59.453+00:00

Hello @KranthiPakala-MSFT and @Vinay5 ,

I have a case where all the year/month/day/time all 4 are dynamin.

Could you please let me in this case?

Thank you.

Accepted answer

KranthiPakala-MSFT 46,442 Reputation points Microsoft Employee

2021-05-04T23:02:51.887+00:00
Hi @Vinay5 ,

Here is the GIF of my pipeline. See if that helps:

Below are screenshots of Getmetadata activity from second/child pipeline in which we get the file names from time folder:

Dataset parameters:

Dataset connection dynamic expression:

GetMetada settings:

Hope this helps.

----------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment

2 additional answers

KranthiPakala-MSFT 46,442 Reputation points Microsoft Employee

2021-05-03T22:38:16.62+00:00

Hi @Vinay5 ,

Thanks for reaching out. As per my understanding I assume that you might be pointing your dataset file path to '/data/' and hence you are just receiving the year folder name.

In order to get the file name list, you will have to point your dataset file path to the sub-folder where the files are actually located.

For example as below:

Which will result all the files inside the subfolder pointed in the file path of your dataset in Getmetadata activity.

In case if you need to pass the year/month/date dynamically then you will have to declare dataset parameter for each (for year, month & date) and then use concatenation function to form the file path in Dataset connection settings. And in GetMetadata activtiy level you can use dynamic expression to get the year, month and date values based on your requirement.

Concatenation expression used in below sample: @concat('input/',dataset().ds_yearFolder,'/',dataset().ds_monthFolder,'/',dataset().ds_dateFolder)

Below is a sample GIF (Please note that in this sample I am passing static values for year, month & date to dataset parameters)

Hope this helps. Do let us know if you have further query.

----------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.

1 person found this answer helpful.
Vinay5 46 Reputation points

2021-05-04T02:48:44.98+00:00

Hi Kranti,

Thanks for your reply.

Actually, my file will be present in the time folder.
For instance, a file will be placed in source path at 17:00 and sometime it will be between 10:00 AM to 14:00 . So, in such cases, there will be one more subfolder after day folder with TIME folder . For example, if the file is placed at 17:00, there will be a folder named 17. And, this 'TIME' folder changes everyday and it is not constant. So, how can i pass this time folder dynamically ,so that my pipeline picks the file from the TIME folder.

KranthiPakala-MSFT 46,442 Reputation points Microsoft Employee

2021-05-04T04:40:29.137+00:00

Hi @Vinay5 ,

Thanks for your reply and additional clarification. In order to get the time folder list for a particular day, you can have parent pipeline in which you use a Getmetadata activity in which you will point your dataset to the date folder and get the list of childItems nothing but the time folder names. and then have a subsequent ForEach activity and pass the output of previous getmetadata activity nothing the list of time folders and iterate through each folder and have an execute pipeline activity inside forEach activity from which you will pass the time folder name as a parameter to child pipeline in which you will have the above mentioned flow. This should help achieve your requirement.

Hope this input helps. Do let us know if you have further query.

Vinay5 46 Reputation points

2021-05-04T04:52:51.103+00:00

Hi Kranthi,

Thanks for your reply.

So, the should I implement solution -2 in conjunction with solution 1 provided above.
Or solution -2 should be implemented seperately. Which pipeline , will the execute activity trigger in this case.?
The time folder varies every day, and the pipeline should automatically search for folder in which file are available.

KranthiPakala-MSFT 46,442 Reputation points Microsoft Employee

2021-05-04T05:04:21.147+00:00

Hi @Vinay5 ,

Yes you are correct, first you will have a parent pipeline (Let say P1) in which you will just get the list of Time folder names dynamically by pointing the dataset to date folder and then followed by a ForEach to iterate through each Time Folder and you will have an Execute pipeline activity inside it. You will pass the time folder name from this execute pipeline activity to child pipeline (Let say P2) as a parameter and that parameter value will be used in the GetMetadata activity dataset to get the file name in it.

Parent pipeline - P1 - is used to get the Time folder names
Child pipeline - P2 - is used to get the file in that Time folder for a particular day.

Vinay5 46 Reputation points

2021-05-04T05:43:43.86+00:00

@KranthiPakala-MSFT
Hi Kranthi,

I got the below child items , when I pointed the input dataset to the day folder.
childItems": [
{
"name": "21",
"type": "Folder".
Then I had a foreach activity in which the items expression is as below.

@activity('Get Metadata1').output.name.

Then I had a execute pipeline inside foreach activity. You mentioned to pass the time folder name from this execute pipeline activity to child pipeline (Let say P2) as a parameter. But, how can I do this. Actually, I should be invoking child pipeline from the parent pipeline.
So, in this case what would be my child pipeline. I got confused in the second part. Could you please kindly clarify this.

Thank you.

Vinay5 46 Reputation points

2021-05-04T12:07:50.093+00:00

@KranthiPakala-MSFT

Hi Kranthi,

I further tried the above solution and almost got the solution, but stuck at the last point.
I was able to get the time folder as input to the second getmetadata activity in the child pipeline. But, I am not able to get the file from the getmetadata activity, the output from getmetadata of the child pipeline doesnt return the filename. Could you please let me know the settings in the child pipeline, including the dataset settings.

Thank you

KranthiPakala-MSFT 46,442 Reputation points Microsoft Employee

2021-05-05T15:16:07.677+00:00

Hi @Vinay5 - Just checking to see if the above solution was helpful to resolve your issue?
Sign in to comment
Vinay5 46 Reputation points

2021-05-05T15:31:36.05+00:00

Hi Kranthi,

Thanks for the detailed explanation. I go the output as required.
Please sign in to rate this answer.
Vinay5 46 Reputation points

2021-05-12T13:46:39.837+00:00

@KranthiPakala-MSFT

Hello Kranthi,

Further to the above requirement, I need to pass all the files present in the time folder as a parameter to a databricks notebook recursively. This notebook is going to parse all the files present in the time folder.
My adf pipeline should be able to pass each file from time folder one after the other as a parameter to this notebook. I have dbutil text command in the notebook. So, basically for a given day there will be 24 time folders and each time folder has different files, so I should pass each file from each time folder as a parameter to my databricks notebook.
Please let me know how this can be achieved.
Thank you.
Sign in to comment

Share via

Accessing file name from nested folder structure and passing it to Notebook activity as a parameter in ADF.

2 additional answers