upsert the data from on premises Mysql to ADLS GEN2(format CSV files)

Rakesh Kumar 45 Reputation points
2023-10-17T19:12:04.49+00:00

I am moving the data from On-Premises SQL server to ADLS in CSV format. I have done the incremental load but don't know how to do the UPSERT in csv files using copy activity.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,489 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,750 questions
{count} votes

Accepted answer
  1. Suba Balaji 11,186 Reputation points
    2023-10-18T04:22:34.0066667+00:00

    Hi Rakesh Kumar

    In ADF we cannot perform upsert to csv files directly. For table based sinks only, in copy activity and data flow you have the upsert option.

    As a workaround, you can have a table in Azure SQL and use that for upsert operation. Once the copy to table completes, you can have another copy activity to copy from table to a csv file.

    For more details on how do we implement upsert in copy activity, please refer to this video and for the data flow you may refer to this video.

    Please let us know if you need any other information or detail regarding this,

    Thank you


0 additional answers

Sort by: Most helpful