question

MaheshMadhusanka-2060 avatar image
0 Votes"
MaheshMadhusanka-2060 asked Sisif0 answered

How Can i Increment Data Append to initial File in Data Factory

Hi Team,

Currently i have copied Single bulk table from SQL database to Datalake 2, But now i'm stuck on the how to increment daily data(insert and Modified) on that bulk file, How can i do? What is the easiest way to do that? Your immediate response much appreciate.


azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered MakotoOda-1289 commented

Hi @MaheshMadhusanka-2060 ,

Welcome to Microsoft Q&A and thank you for your query.

Unfortunately appending to an existing blob using ADF copy activity is not supported. I would recommend you to please up-vote and/or comment on the below existing similar feature request from other users. Up-voting/commenting will help to increase the priority of the feature request.

Existing feature request: https://feedback.azure.com/forums/270578-data-factory/suggestions/36163726-append-replace-upsert-the-outgoing-data-from-copy

Additional info:

Please have a look at this SO thread and see if that helps as a workaround for your scenario: Is there any workaround for using Append blob in Azure Data Factory copy?


Hope this info helps.



Thank you

Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.




· 8
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @KranthiPakala-MSFT ,


Thank you for your response and info, You mean we don't have any option to new incremental data append on the exsisting bulk table csv file on azure data lake? is it?


0 Votes 0 ·

Hi MaheshMadhusanka-2060,

Thanks for your response. Yes, append data to an existing file is not support by ADF copy at this time. Please feel free to up-vote/comment on the above feature request thread to increase the priority of the feature request.

0 Votes 0 ·

Hi @KranthiPakala-MSFT

Thank you for your response, Yes I already Vote it, However Can i use Union function on data flow or is there have a any merge feature on that, for this concern i have already raised new Q & A Can you please advise on this as well?

https://docs.microsoft.com/en-us/answers/questions/32743/how-to-setup-copy-activity-with-merge-data-flow-in.html

0 Votes 0 ·
Show more comments

@KranthiPakala-MSFT The feature request link which you wrote seems to be not found currently. Could you please tell us the current link?

0 Votes 0 ·

Recently the feedback forum has been migrated to Ideas forum: Please find the latest feedback link here : https://feedback.azure.com/d365community/idea/47f287c9-6d26-ec11-b6e6-000d3a4f032c

Thank you

0 Votes 0 ·

@KranthiPakala-MSFT Thank you for your answer. I understand.

0 Votes 0 ·
Sisif0 avatar image
0 Votes"
Sisif0 answered

Hi @MaheshMadhusanka-2060

I loading constantly new data and I'm storing it in different files, for instance: File_2022-05-11_21-57.parquet.

For working with all data stored in these files, I have created an external table in Synapse SQL serverless. This external table is reading all files in some folder with filename equal to: File_*.parquet

That's the way I found to work query them, etc.

Regards

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.