Dynamically Select the files

Question

Dynamically Select the files

Rohit Kulkarni 731

Hello Team,

I need to select the files from this folder.Due to timestamp i am not able to do dynamically.IS there any way where i can select only those 3 files and run dynamically

File Path:outbound/em_dataextracts

User's image

Please advise

Rohit

Chandra Boorla 15,475 Reputation points Microsoft External Staff Moderator

2025-01-09T18:29:52.5+00:00
Hi @Rohit Kulkarni

Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

To dynamically select files based on their last modified date in Azure Data Factory, you can use the filter functionality in your pipeline. You can set up a copy activity where you specify the last modified date range in the advanced settings. For example, you can use a dynamic expression to filter files that have been modified within a specific time frame.

Here is a general approach you can follow:

In your copy activity, go to the source settings --> Under the "Advanced" section, find "Filter by last modified." --> Set Dynamic Expressions - Use the ADF dynamic content feature to specify the time range for filtering files.

For example:

Start Time:

@formatDateTime(addMinutes(pipeline().TriggerTime, -5), 'yyyy-MM-ddTHH:mm:ssZ')

This expression selects files modified within the last 5 minutes.

End Time:

@formatDateTime(pipeline().TriggerTime, 'yyyy-MM-ddTHH:mm:ssZ')

This expression will filter files modified in the last 5 minutes. Adjust the time frame as needed to match the specific files you want to select.

This way, you can dynamically select files based on their last modified date without manually specifying each file.

Important Considerations:

Time Zones - Be mindful of time zones. ADF uses UTC internally. If your source system uses a different time zone, you might need to adjust your expressions accordingly using functions like convertFromUtc or convertToUtc.

Data Types - Ensure the date/time format in your expressions matches what ADF expects. Inconsistencies can lead to errors.

Performance - For very large datasets, filtering by last modified date can still be resource-intensive. Consider partitioning your data or using other optimization techniques if necessary.

For more details, please refer the following documentations:

https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-incremental-copy-files-last-modified-date#configure-a-data-pipeline-for-incremental-copy

https://learn.microsoft.com/en-us/azure/data-factory/solution-template-copy-new-files-last-modified-date#how-to-use-this-solution-template

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

I hope this information helps. Please do let us know if you have any further queries.

Thank you.
Rohit Kulkarni 731 Reputation points

2025-01-10T16:04:34.79+00:00

In copy Activity in target it will select only 1 file table and load.It will not select the dynamically the files and load the data into recepective table
Rohit Kulkarni 731 Reputation points

2025-01-10T16:06:00.51+00:00

I tried with copy Activity but not able to select the respective tables in target dynamically how it can be done
Chandra Boorla 15,475 Reputation points Microsoft External Staff Moderator

2025-01-11T02:43:25.32+00:00

@Rohit Kulkarni

Please refer the following documentation for a valuable insight:

https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-lastmodified-copy-data-tool

If the issue still persists, please do let us know. Glad to help.

Thank you.
Chandra Boorla 15,475 Reputation points Microsoft External Staff Moderator

2025-01-13T06:20:48.2033333+00:00

@Rohit Kulkarni

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Rohit Kulkarni 731 Reputation points

2025-01-13T11:32:48.79+00:00

I am using Get data and extracting the file content in the folder

I need only type='file'.But it is selecting folder also.Please refer the screenshot below
Chandra Boorla 15,475 Reputation points Microsoft External Staff Moderator

2025-01-16T20:42:15.59+00:00

@Rohit Kulkarni

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Chandra Boorla 15,475 Reputation points Microsoft External Staff Moderator

2025-01-17T17:25:20.99+00:00

@Rohit Kulkarni

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Your answer

Rohit Kulkarni 731 Reputation points

2025-01-10T16:04:34.79+00:00

In copy Activity in target it will select only 1 file table and load.It will not select the dynamically the files and load the data into recepective table
Rohit Kulkarni 731 Reputation points

2025-01-10T16:06:00.51+00:00

I tried with copy Activity but not able to select the respective tables in target dynamically how it can be done
Chandra Boorla 15,475 Reputation points Microsoft External Staff Moderator

2025-01-11T02:43:25.32+00:00

@Rohit Kulkarni

Please refer the following documentation for a valuable insight:

https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-lastmodified-copy-data-tool

If the issue still persists, please do let us know. Glad to help.

Thank you.
Chandra Boorla 15,475 Reputation points Microsoft External Staff Moderator

2025-01-13T06:20:48.2033333+00:00

@Rohit Kulkarni

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Rohit Kulkarni 731 Reputation points

2025-01-13T11:32:48.79+00:00

I am using Get data and extracting the file content in the folder

I need only type='file'.But it is selecting folder also.Please refer the screenshot below
Chandra Boorla 15,475 Reputation points Microsoft External Staff Moderator

2025-01-16T20:42:15.59+00:00

@Rohit Kulkarni

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Chandra Boorla 15,475 Reputation points Microsoft External Staff Moderator

2025-01-17T17:25:20.99+00:00

@Rohit Kulkarni

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

@Rohit Kulkarni

To address the issue of selecting only files and excluding folders while using the "Get Metadata" activity in Azure Data Factory, you can use the following approach:

Use the Get Metadata Activity - Get Metadata Activity, this activity can be used to retrieve the properties of the files in your folder. Field List, in the Get Metadata activity, specify the field list to include childItems. This will return a list of items in the specified folder.

Filter the Results - Filter Activity, after retrieving the metadata, use a Filter activity to filter out only the items of type 'file'. Expression, you can use an expression like the following to filter the results - @equals(item().type, 'file')

Use the Filtered Output in Your Copy Activity - Copy Activity, connect the output of the Filter activity to your Copy activity. Dynamic Content, in the source settings of the Copy activity, use the filtered list of files to dynamically select the files you want to copy.

Important Considerations:

Performance - Ensure that your folder does not contain an excessive number of files, as this could impact performance.

Testing - Test the pipeline with a smaller dataset to ensure that the filtering works as expected before scaling up.

I hope this information helps.

Thank you.

Share via

Dynamically Select the files

1 answer

Your answer