SAP latency data

Vineet S 205 Reputation points
2024-04-23T07:29:53.95+00:00

Hi Expert,

how to we can load the data from modified data in updated or insert fields in databricks using ADF or data bricks on trigger level instead of loading multiple times

example:

table updated or inserted with new records how table change and change table will identified.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,977 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,774 questions
{count} votes

Accepted answer
  1. AnnuKumari-MSFT 31,726 Reputation points Microsoft Employee
    2024-04-23T17:51:50.95+00:00

    Hi Vineet S ,

    Thankyou for using Microsoft Q&A platform and thanks for posting your query here .

    Based on the your query, it seems like you are looking for a way to load incremental data (to avoid full load everytime) in SAP using ADF or databricks.

    There could be multiple ways to achieve this requirement , either creating datapipelines, or write databricks code , or using mapping dataflow, or else by using CDC capabilities within ADF, which seems to be the most efficient approach.

    Change data capture (CDC) allows you to track changes to a table and capture only the modified data, which can help to reduce the amount of data that needs to be processed and loaded.

    For implementation, kindly check the below relevant resources:

    Overview and architecture of the SAP CDC capabilities

    SAP CDC in Azure Data Factory

    Hope it helps. Thankyou

    0 comments No comments

0 additional answers

Sort by: Most helpful