Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,826 questions
csv incremental data load to database concern
Mahesh Madhusanka
106
Reputation points
Hi Team,
Currently, We have requirement to datalake csv file sync with the on-prime database, according to i have created connection and currently working on loading data from datalake csv(one csv file as a table) file to on-prime database table. In the dataLake side csv file loading incremental data with new data growth, According to i want to develop a setup that same as DL csv i want to load increment same data to database table as well, How Can we do that, Could you please guide on that? Your Support much appreciate?
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT 10,136 Reputation points2020-08-24T11:29:03.287+00:00 Hi @Mahesh Madhusanka ,
Welcome to Microsoft Q&A Platform. Thanks for posting the query.
Please share more details to help us understand on the requirement of incremental data load into database table. Below are 2 possibilities as per my understanding.
- If it is loading according to a date column in csv file comparing it with database table date, it can be implemented using following transformations in "Mapping Dataflow".
- If it is loading in "Update and insert" mode into database table(SQL), there is an option as below in sink.
- If it is loading according to a date column in csv file comparing it with database table date, it can be implemented using following transformations in "Mapping Dataflow".
-
Mahesh Madhusanka 106 Reputation points
2020-08-24T17:27:53.203+00:00 Currently i have used below dataflow for incremental data load to append on current bulkfile inside the datalake, According to base on the source data source growth Bulk CSV file growth with the daily incremental data, Now i want to sync csv Data table to database and specially csv file daily incremental data load to database as well? Could you please guide on that?
-
HarithaMaddi-MSFT 10,136 Reputation points
2020-08-26T06:40:03.733+00:00 Hi @Mahesh Madhusanka ,
Thanks for sharing additional details. Please suggest if below options help to load the data into both csv and database to keep them in sync.
- Adding another sink to filter activity to copy the incremental data to database table as well
- Adding copy activity after dataflow is completed in the pipeline
-
HarithaMaddi-MSFT 10,136 Reputation points
2020-08-28T05:28:12.317+00:00 Hi @Mahesh Madhusanka ,
We have not received a response from you. Please let us know for further queries and we will be glad to assist.
-
HarithaMaddi-MSFT 10,136 Reputation points
2020-08-31T07:52:06.883+00:00 Hi @Mahesh Madhusanka ,
Following up to check if you found a solution. Please let us know above details to assist further.
Sign in to comment
Sign in to answer