Compare csv or parquet files,identify delta records

Question

Hi Guys,

We load files into Azure Data lake store every few hours, we want to compare present file with previous file and able to identify delta like merge statement and generate new file, with primary key and action type flag as below

0- insert
1-update
2-delete

Something like cdc on file data, any thoughts?

Bear in mind no database load

Regards,
Navin

Answer 1

See if this helps: https://www.youtube.com/watch?v=GACpvMjOJgE