ADF-Dataflow Table sync records not processed after 12 hrs

Question

ADF-Dataflow Table sync records not processed after 12 hrs

Aaron Wong 40

I have a pipeline which uses a dataflow with SQL CDC feature enabled. I am syncing tables between one Azure SQL to another Azure SQL DB. Since yesterday the pipeline is failing after running for 12 hrs. A particular table dataflow activity is getting timed out. I ran the following queries manually and I do get a response immediately. Not sure why there are no rows processed in the dataflow activity. There are other pipelines from the same source and destination databases that are running fine. Please let me know what could be the reason for this. I tried to run the pipeline again and even after 9 hrs there was no rows processed.

Pipeline Id: 5f6b7d00-339e-4469-93a9-6f63bad74061

Activity Id: 0c1ba17a-e44b-4514-8982-2db9eefc3efa

SELECT TOP 10 * FROM cdc.fn_cdc_get_net_changes_dbo_xxx

(

sys.fn_cdc_get_min_lsn('dbo_xxx'),

sys.fn_cdc_get_max_lsn(),

'all'

)

SELECT sys.fn_cdc_get_min_lsn('dbo_xxx') AS MinLSN,

   sys.fn_cdc_get_max_lsn()              AS MaxLSN;

Venkat Reddy Navari 5,840 Reputation points Microsoft External Staff Moderator

2025-08-26T12:34:37.8433333+00:00
Hi Aaron Wong From what you described, the most likely reason is that the Dataflow is stuck on an LSN range where there are no valid changes. Even though fn_cdc_get_net_changes returns quickly when you run it manually, the activity may still be using an older MinLSN that has already been cleaned up.

Check the MinLSN/MaxLSN values against the CDC change table for that capture instance. If the MinLSN is older than the current start LSN, the Dataflow won’t process anything. In that case, stop and restart the pipeline so it picks up the latest LSN.

Validate the capture instance with sys.sp_cdc_help_change_data_capture to ensure it’s healthy.

Test the same query in a Copy Activity if it succeeds, the issue is isolated to Dataflow.

If this table has very large transactions or if your IR is under-provisioned, scale the IR or enable partitioning on the source to avoid long stalls.

Since your other pipelines are working fine, this really points to a table-specific CDC range issue rather than a platform-wide problem.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

1 answer

Your answer

Venkat Reddy Navari 5,840 Reputation points Microsoft External Staff Moderator

2025-08-26T12:34:37.8433333+00:00

Hi Aaron Wong From what you described, the most likely reason is that the Dataflow is stuck on an LSN range where there are no valid changes. Even though fn_cdc_get_net_changes returns quickly when you run it manually, the activity may still be using an older MinLSN that has already been cleaned up.

Check the MinLSN/MaxLSN values against the CDC change table for that capture instance. If the MinLSN is older than the current start LSN, the Dataflow won’t process anything. In that case, stop and restart the pipeline so it picks up the latest LSN.

Validate the capture instance with sys.sp_cdc_help_change_data_capture to ensure it’s healthy.

Test the same query in a Copy Activity if it succeeds, the issue is isolated to Dataflow.

If this table has very large transactions or if your IR is under-provisioned, scale the IR or enable partitioning on the source to avoid long stalls.

Since your other pipelines are working fine, this really points to a table-specific CDC range issue rather than a platform-wide problem.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Answer 1

Aaron Wong 40

Venkat Reddy Navari, We see the following timestamps in the for the min and max lsn values in the _CT table. "SELECT MIN($start_lsn) AS MinStartLSN_InTable, MAX($start_lsn) AS MaxStartLSN_InTable FROM cdc.dbo_YourTableName_CT;"

User's image

When we run the following query , there are many rows changed.

SELECT COUNT(*) AS ChangeCount FROM cdc.dbo_YourTableName_CT

The count is 42204430. We increased the timeout period to 15 hrs from the default setting and also changed the compute size to 'medium' , after triggering the pipeline to process manually we are getting a new error as shown below- Please suggest to to fix this error. User's image

. Thanks!

Smaran Thoomu 35,285 Reputation points Microsoft External Staff Moderator

2025-08-29T16:18:17.47+00:00
Aaron Wong Thanks for sharing the additional details and screenshots. Based on your update, a few things stand out:

Large Change Volume – Your CDC change table (cdc.dbo_YourTableName_CT) shows 42M+ rows pending. Even though queries return quickly in SSMS, Dataflow has to fetch, serialize, and move that volume across the IR, which is much heavier.

Timeout/IR Scaling – You already increased the timeout to 15 hrs and moved to a Medium IR. With such a large change backlog, the activity may still stall. Try either:

Scaling up further (e.g., Large or XL IR) and/or enabling partitioned reads in Dataflow.

Breaking the ingestion into smaller batches using the start_lsn / end_lsn parameters manually to chunk the workload.

Error in new run – The error shown in your screenshot (Insufficient number of arguments supplied for the procedure or function) typically happens if the Dataflow’s CDC function call doesn’t align with the expected parameters. This can occur when the Dataflow retries with an older MinLSN that is already invalid or cleaned up. Resetting the pipeline’s state (to pick up from the current valid MinStartLSN) often resolves it.

Next steps to try

Run sys.sp_cdc_help_change_data_capture to verify the capture instance health.

Compare MinStartLSN_InTable with the pipeline’s last checkpoint. If the pipeline is stuck on an invalid LSN, reset/restart it from the current MinLSN.

If reprocessing all 42M changes in a single Dataflow run is not practical, consider a staged Copy Activity or batching with ADF instead of Dataflow CDC for this table.

Since your other CDC pipelines are working fine, this really looks like a table-specific backlog + checkpoint misalignment issue.

Let us know if resetting the CDC state or chunking the ingestion reduces the stall.
Venkat Reddy Navari 5,840 Reputation points Microsoft External Staff Moderator

2025-09-01T10:13:46.0066667+00:00

Aaron Wong Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

ADF-Dataflow Table sync records not processed after 12 hrs

1 answer

Your answer