How to pass directory dynamically from the config file

Question

Hi Team, I am developing a pipeline that will copy the files from one location to another location within blob storage. I have created a config file that will have details of the filename, sourcelocation, and destination location. Config file: I am doing a lookup on the config file then filter the rows according to the requirement and then for each activity which will loop through based on the filter value.

I have 2 files in the location1 and the output of the filter activity is
{ "ItemsCount": 3, "FilteredItemsCount": 2, "Value": [ { "ID": "1", "FileName": "File1_YYYYMMDDHHMISS.txt", "SourceLocation": "Location1", "DestinationLocation": "Location2", "ActiveFlag": "Y" }, { "ID": "2", "FileName": "File2_YYYYMMDDHHMISS.txt", "SourceLocation": "Location1", "DestinationLocation": "Location2", "ActiveFlag": "Y" } ] } Inside the foreach activity, I am looping through each file get the SourceLocation and DestinationLocation and pass it to copy activity from the above output of filter activity

SrcLake_Filename expression - @concat(substring(item().FileName,0,lastindexof(item().FileName,'_')),'*'). The pipeline is getting succeeded but the files are not getting copied over to the destination. Appreciate your help Thanks, Jaganath

Answer 1

Hi Jaganath Kumar ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query here.

As per my understanding, you are trying to copy the files dynamically by making use of wildcard file path and storing the source file names in a config file. However, the results are not coming as expected. Pipeline are succeeding, however, it is not copying the file.

Could you please share what expression are you using in filter activity and check what is the output of filter activity ?

Additionally, kindly try to perform data preview in the source , provide the value for source location and fileName for one of the entries and see what is coming as the output of data preview. If it is also blank , or not.

I tried your scenario from my end and it was working perfectly fine. Please share the above details. Thankyou