Data Factory/Synapse copy data activity source has a feature to point to a text file that lists each file that we want to copy to the sink. The functionality works great but I'm breaking my head as to how I can generate that text-file in the first place using the files in the blob storage. It worked great because I created the file list manually and uploaded to the blob but that ain't going to work in end-2-end flow.
In the past, I've written shell script to generate the file-list and executed it before the session/mapping that does the actual load to staging tables etc (you know which ETL tool I'm talking about) but how can we do it in the Azure ADF landscape?
Thinking of leveraging get metadata activity on the container, looping through each and inserting into a database. Then having a stored proc to group them into respective "file-list" but how can I make ADF create a blob storage file with list of files in that? Another option is to merge all files using the same metadata activity but this seems to me like a simple feature and I don't mean to beat a dead horse, but I still don't have a clear design path for this.
Any guidance is greatly appreciated. It seems to me like a simple feature.