Hi @PeterSh ,
Thank you for posting query in Microsoft Q&A Platform.
To summarize your ask, its more about best way to find only new rows form parquet file to process them to SQL table. Please correct me if I am wrong.
You could consider either of below approaches if that helps.
- Save your new rows every time as a separate file with some datetime in file name. So that we can take only that file and process it in to SQL Table.
- Consider having some flag column or date Time column which tells us that these are new rows. So that we can use source transformation for our source file and then Filter transformation to filter out new rows based on flag or datetime value.
Hope this helps. Please let us know how it goes. Thank you.