How to read Data Partitioned files from ADLS using ADF mapping data flow source options

Question

Hi All,

Good Day!!

We have a requirement to join multiple files which are available in ADLS folders which we have ingested from SQL Source and derive business logic accordingly. Since we store the Table data in ADLS with table name and date partitioned folder for incremental load from Source System. Now in order to read multiple files and join like SQL Query, initially I have to read date partitioned files in mapping data flow dynamically as per the current date.

ADLS folder structure: Tenant1/Raw/PolicyAdmin/SRC_ConceptOne/Entity/2021/07/09/

Can any give me idea how to read date partitioned files dynamically in ADF mapping data flow.

Thank you in anticipation!!

Regards,
Mahammad Khan

Answer 1

Hi @Abdulla Mahammad Khan Dawood ,

Thank you for posting query in Microsoft Q&A Platform. Kindly check below detailed example, In which I am using wildcard to copy all dates files for "07" month folder.

Folder structure I had on my ADLS demofiles/yyyy/MM/dd/emp_yyyyMMdd.csv

Source Transformation in dataflow to copy all .csv files from all dates folders(01,02,03)

Add sink transformation after above step to load data in to your Target storage.

Please check below documentation to know more about Wild card file paths in Source Transformations.
https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-data-lake-storage#source-transformation

Hope this will help. Thank you.

----------------------------------

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
• Want a reminder to come back and check responses? Here is how to subscribe to a notification.