SAP latency data

Vineet S 165

Hi Expert,

how to we can load the data from modified data in updated or insert fields in databricks using ADF or data bricks on trigger level instead of loading multiple times

example:

table updated or inserted with new records how table change and change table will identified.

AnnuKumari-MSFT 31,061 Reputation points Microsoft Employee

2024-04-23T17:51:50.95+00:00

Hi Vineet S ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query here .

Based on the your query, it seems like you are looking for a way to load incremental data (to avoid full load everytime) in SAP using ADF or databricks.

There could be multiple ways to achieve this requirement , either creating datapipelines, or write databricks code , or using mapping dataflow, or else by using CDC capabilities within ADF, which seems to be the most efficient approach.

Change data capture (CDC) allows you to track changes to a table and capture only the modified data, which can help to reduce the amount of data that needs to be processed and loaded.

For implementation, kindly check the below relevant resources:

Overview and architecture of the SAP CDC capabilities

SAP CDC in Azure Data Factory

Hope it helps. Thankyou
Vineet S 165 Reputation points

2024-04-23T22:34:56.79+00:00

How it will know the new data is available :)