How to read Last Partitioned File from ADLS using ADF mapping Data Flow
we have a requirement to read last ingested partitioned parquet files which are available in ADLS folders.
ADLS folder structure: XXX/processed/year=yyyy/month=MM/day=dd
Can any give me idea how to read last date partitioned files dynamically in ADF mapping data flow. I mean how to recover partitioned folders and select last in time.
Did you ever come to a simple resolution to this?
I'm after the same requirement and this is no where near as simple as I had hoped.
My thoughts at the moment are leaning towards the creation of a trigger/metadata file in ADLS when the source file(s) are landed in ADLS - which contains the YYYY/MM/DD datepart file path in it - so the file will always contain the most recent file path. From there, the subsequent dataflow pipeline can then read the trigger file, fetch the datepart file path in it and then read from it.
For us the recommended solution by MSFT didn't fit the brief.
Sign in to comment