Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Azure SAP CDC Connector - Inconsistent count from SAP BW source vs Ingested Data using Data Factory
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 12%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESF%3C/text%3E%3C/svg%3E)
Sondon, Fritz Joery
0
Reputation points
Hi Microsoft,
We'd like to ask your confirmation if this is the default behavior of SAP CDC connector. We are pulling data from SAP BW using Data Factory and ingest it to Data Lake. However, we noticed that counts are not consistent in tables from SAP BW(Source) to what has been ingested in Data Lake.
Below shows different Record count per table. The total amount is the same and no issue.
Can we kindly know the reason behind this behavior and if it is indeed the default behavior of SAP CDC Connector?
Thank you.
Azure Data Factory
{count} votes
-
AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator2023-02-27T19:04:27.41+00:00 Hi Sondon, Fritz Joery ,
Welcome to Microsoft Q&A platform and thanks for posting your question here.
As I understand your query, you are trying to load data from SAP BW source to ADLS . However, you are experiencing some inconsistency in the record count of source vs sink. Please let me know if that is not the correct understanding.
Could you please share more details of your pipeline with screenshot of source , sink and settings tab along with the details of copy activity run by clicking the below highlighted icon . Have you enabled fault tolerance in the copy activity?
-
Sondon, Fritz Joery 0 Reputation points
2023-03-01T07:55:34.1766667+00:00 Hi AnnuKumari,
Yes that is correct. We are using SAP CDC connector and did not use Copy Activity but using Dataflow. Run mode is set to "Full on the first run, then incremental".
We followed the steps described in the documentation below to implement SAP CDC in Data Factory.
In Dataflow, we have 3 activities:
- SAP - use SAP CDC Link Service and connected to SAP BW and read data.
- AdditionalMetadataColumns - additional columns
- RawDataLake - sink
Below is the result of other ingested table which has consistent # of records read until sink, however, checking the actual record counts from SAP BW is different to the counts extracted.
The total amounts remains the same with SAP BW source vs extracted in ADLS and has no issues.
May we know if this is expected behavior and what cause behind this behavior or probably we missed something in the setup?
Please let me know if we need more details. Thank you for your help.
-
Sondon, Fritz Joery 0 Reputation points
2023-03-03T12:28:07.6333333+00:00 May I know if you have update about our inquiry? Thank you for your assistance.
Sign in to comment
Sign in to answer