How to pass directory dynamically from the config file

Jaganath Kumar 40 Reputation points
2024-02-13T06:23:48.6766667+00:00

Hi Team, I am developing a pipeline that will copy the files from one location to another location within blob storage. I have created a config file that will have details of the filename, sourcelocation, and destination location. Config file: User's image I am doing a lookup on the config file then filter the rows according to the requirement and then for each activity which will loop through based on the filter value. User's image

I have 2 files in the location1 and the output of the filter activity is
{ "ItemsCount": 3, "FilteredItemsCount": 2, "Value": [ { "ID": "1", "FileName": "File1_YYYYMMDDHHMISS.txt", "SourceLocation": "Location1", "DestinationLocation": "Location2", "ActiveFlag": "Y" }, { "ID": "2", "FileName": "File2_YYYYMMDDHHMISS.txt", "SourceLocation": "Location1", "DestinationLocation": "Location2", "ActiveFlag": "Y" } ] } Inside the foreach activity, I am looping through each file get the SourceLocation and DestinationLocation and pass it to copy activity from the above output of filter activity User's image

SrcLake_Filename expression - @concat(substring(item().FileName,0,lastindexof(item().FileName,'_')),'*'). The pipeline is getting succeeded but the files are not getting copied over to the destination. Appreciate your help Thanks, Jaganath

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,067 questions
{count} votes

Accepted answer
  1. AnnuKumari-MSFT 29,586 Reputation points Microsoft Employee
    2024-02-14T02:49:28.97+00:00

    Hi Jaganath Kumar ,

    Thankyou for using Microsoft Q&A platform and thanks for posting your query here.

    As per my understanding, you are trying to copy the files dynamically by making use of wildcard file path and storing the source file names in a config file. However, the results are not coming as expected. Pipeline are succeeding, however, it is not copying the file.

    Could you please share what expression are you using in filter activity and check what is the output of filter activity ?

    Additionally, kindly try to perform data preview in the source , provide the value for source location and fileName for one of the entries and see what is coming as the output of data preview. If it is also blank , or not. User's image

    I tried your scenario from my end and it was working perfectly fine. Please share the above details. Thankyou

    0 comments No comments

0 additional answers

Sort by: Most helpful