Thanks for the question and using MS Q&A platform.
To process only the files matching the pattern .\\source\\SystemA_*.xml
within a single data flow or pipeline without modifying the dataset configuration daily, you can use dynamic content in your data flow or pipeline configuration. Here’s how you can achieve this:
Dynamic Content in Data Flow:
- In your data flow, create a parameter (let’s call it
SourceFilePath
) that represents the folder path where your XML files are located (e.g.,.\\source
). - Use this parameter in your source dataset configuration. For example, if you’re using a
File System
source, set the folder path to@{dataset().SourceFilePath}
. - In your data flow activities, use the
SourceFilePath
parameter to dynamically read files matching the pattern.\\source\\SystemA_*.xml
.
Dynamic Content in Pipeline:
- Create a pipeline parameter (e.g.,
SourceFolderPath
) representing the folder path where your XML files reside (e.g.,.\\source
). - In your pipeline, use this parameter in the
ForEach
activity to iterate over files matching the pattern.\\source\\SystemA_*.xml
. - Inside the
ForEach
activity, configure your data flow activity to read the current file (using@item().name
or@concat(parameters('SourceFolderPath'), '/', item().name)
).
By using dynamic content, you avoid hardcoding the file names and adapt to changes in the folder structure without modifying the dataset configuration daily
Hope this helps. Do let us know if you any further queries.