question

NavinDondapati avatar image
0 Votes"
NavinDondapati asked NavinDondapati commented

Compare csv or parquet files,identify delta records

Hi Guys,

We load files into Azure Data lake store every few hours, we want to compare present file with previous file and able to identify delta like merge statement and generate new file, with primary key and action type flag as below

0- insert
1-update
2-delete

Something like cdc on file data, any thoughts?

Bear in mind no database load

Regards,
Navin

azure-data-factoryazure-data-lake-storage
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

MarkKromer-MSFT avatar image
1 Vote"
MarkKromer-MSFT answered NavinDondapati commented
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you, exactly what I am looking for

0 Votes 0 ·

Mark link is the answer

0 Votes 0 ·