Data factory CDC empty File creation

Question

Data factory CDC empty File creation

Franklin John 25

We have created CDC in Azure to update data from SQL to the data lake It's working correctly whenever I update in SQL it creates a file in the data lake. But sometimes it creates empty files how to avoid those empty file creation ADF change data capture (preview)

Answer accepted by question author

0 additional answers

Your answer

Answer 1

AnnuKumari-MSFT 34,566 Microsoft Employee Moderator

Hi Franklin John ,

Welcome to Microsoft Q&A platform and thanks for posting your question here.

As I understand your query, you are trying to perform change data capture in ADF for incremental records but it is creating empty files for you. Please let me know if that is not the correct understanding.

I tried to repro your scenario by creating CDC with real time latency , however, I didn't get the same issue. Kindly share what latency have you set and have you enabled CDC in source table before the records were inserted?

Observation : If your source table already had data before you enabled CDC in the source table in SQL db , then it is prone to create empty file in ADLS for the first run . Make sure to enable the CDC first and then insert records in the source table and use CDC capability from ADF.

Hope it helps. Please revert back with any follow up query if you have , I can take it forward to internal team . Kindly accept the answer as accepted answer helps community as well. Thankyou

Franklin John 25 Reputation points

2023-05-19T15:15:43.7133333+00:00

I have Enabled CDC for two tables in SQL and I modified only one data in one table. but it creates so many files. I have attached screenshots for both tables' sink paths. I received the changes that the file size is 13.81 kb. All other 822kb file is empty files. I don't know why those files created
AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2023-05-19T15:42:17.93+00:00

Hi Franklin John ,

I can also see lots of files got created in my folder as well. Taking this forward to internal team for more details.
Franklin John 25 Reputation points

2023-05-19T15:49:10.0866667+00:00

Thanks for the Update. Waiting for your solution. It's a critical one for me because I need to trigger another pipeline based on file creation. But here it triggers another pipeline unwantedly
Namindu Sanchila 0 Reputation points

2023-05-19T16:06:53.6433333+00:00

Commenting for the follow up!
AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2023-05-19T16:13:57.5033333+00:00

Hi Franklin John ,

I have taken it forward to the internal team. I will keep you posted once I hear back from them . Thankyou for your patience.
AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2023-05-24T09:30:07.7333333+00:00

Hi Franklin John ,

It has been taken into consideration as a bug and the fix has been released. It would take a week's time to get deployed. Kindly verify from your end after a week and keep us updated. Thankyou
Namindu Sanchila 0 Reputation points

2023-05-24T10:37:59.08+00:00

@AnnuKumari-MSFT , Commenting on behalf of Franklin John,

Please, let us know once its deployment is completed.
AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2023-06-03T01:20:58.1633333+00:00

Hi Namindu Sanchila / Franklin John ,

Could you please check if you are still seeing empty CDC files ? It seems the deployment has been rolled out for multiple regions. I don't see any empty files at my end now. Kindly confirm if everything looks good, requesting you to accept the answer by clicking on Accept answer button.
AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2023-06-06T09:43:27.9233333+00:00

Hi Franklin John ,

Thank you for marking this answer as accepted. Please feel free to take a survey on the relevant answer. You can help us improve by leaving feedback verbatim for the support you got from the platform.

Thankyou

Share via

Data factory CDC empty File creation

0 additional answers

Your answer