Hello @Nofar Hameiri
To dynamically construct the desired datasets and process them in the data flow, you can use the following 7-steps:
- Create a pipeline with a Lookup activity to read the control file. In the Lookup activity, set the file path to the control file and the file format to DelimitedText. In the DelimitedText format settings, set the delimiter to "|" and the first row as header.
- Add a Filter activity after the Lookup activity to filter the rows based on the file name prefix. In the Filter activity, set the condition to
startsWith(item().FILENAME, "providers")
for the provider dataset andstartsWith(item().FILENAME, "network")
for the network dataset. - Add a ForEach activity after the Filter activity to iterate over the filtered rows. In the ForEach activity, set the Items property to
@activity('Filter').output.value
to iterate over the filtered rows. - Inside the ForEach activity, add a Copy activity to copy each file into its correlated dataset. In the Copy activity, set the source dataset to the file path and the sink dataset to the corresponding dataset based on the file name prefix. You can use dynamic content to set the dataset name based on the file name prefix.
- Add a Wait activity after the Copy activity to wait for the file to be fully copied before processing it in the data flow.
- Add a Data Flow activity after the Wait activity to process the data in the datasets. In the Data Flow activity, you can use the source dataset and sink dataset that you created in the Copy activity.
- Run the pipeline to process the data in the datasets.
By using these above steps, you can dynamically construct the desired datasets and process them in the data flow each time the control file is added.
I hope that this response has addressed your query and helped you overcome your challenges. If so, please mark this response as Answered. This will not only acknowledge our efforts, but also assist other community members who may be looking for similar solutions.