Share via

Mapping data flow to implement an incremental loading for a SQL table

pmscorca 1,052 Reputation points
2023-11-19T17:53:10.0766667+00:00

Hi,

I'm searching a good example about to implement a mapping data flow in order to accomplish an incremental loading for a SQL table, where the source is a SQL table in a staging area on a Synapse SQL pool and the sink is a SQL table in a consolidation area. I'd like to detect first which rows to update and then which rows to insert as for a SCD1 (not a SCD2, I don't need to create a new row to historicize the changes respect to a natural o business key and I don't need to manage a validity period between a start date and an end date).

Any helps to me, please? Many thanks!

Azure Synapse Analytics
Azure Synapse Analytics

An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.


1 answer

Sort by: Most helpful
  1. Subashri Vasudevan 11,306 Reputation points Volunteer Moderator
    2023-11-20T03:57:56.79+00:00

    Hi,

    For scd-1 implementation, please check the below videos with only difference being source type. You can still follow the steps for SQL source

    Video#1

    Video#2

    Please try and let us know if you have questions.

    Thanks

    Was this answer helpful?

    1 person found this answer helpful.

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.