How Can i Increment Data Append to initial File in Data Factory

Question

Hi Team,

Currently i have copied Single bulk table from SQL database to Datalake 2, But now i'm stuck on the how to increment daily data(insert and Modified) on that bulk file, How can i do? What is the easiest way to do that? Your immediate response much appreciate.

Answer 1

Hi @Mahesh Madhusanka ,

Welcome to Microsoft Q&A and thank you for your query.

Unfortunately appending to an existing blob using ADF copy activity is not supported. I would recommend you to please up-vote and/or comment on the below existing similar feature request from other users. Up-voting/commenting will help to increase the priority of the feature request.

Existing feature request: https://feedback.azure.com/forums/270578-data-factory/suggestions/36163726-append-replace-upsert-the-outgoing-data-from-copy

Additional info:

Please have a look at this SO thread and see if that helps as a workaround for your scenario: Is there any workaround for using Append blob in Azure Data Factory copy?

Hope this info helps.

----------

Thank you

Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.

Answer 2

Hi @Mahesh Madhusanka

I loading constantly new data and I'm storing it in different files, for instance: File_2022-05-11_21-57.parquet.

For working with all data stored in these files, I have created an external table in Synapse SQL serverless. This external table is reading all files in some folder with filename equal to: File_*.parquet

That's the way I found to work query them, etc.

Regards