Within my Sink of my Data Flow it took 3 minutes 16 seconds to write 106 rows. How do I reduce this time to write rows to a reasonable level?

Question

Within my Sink of my Data Flow it took 3 minutes 16 seconds to write 106 rows. How do I reduce this time to write rows to a reasonable level?

Kieran Wood 71

Within my Delta Sink of my Data Flow it took 3 minutes 16 seconds to write 106 rows. How do I reduce this time to write rows to a reasonable level?

The data source is of type SAP CDC and has 20 million rows and is managing deltas correctly. The source activity executed in a more reasonable 21 seconds.

The integration runtime is...

Compute Size: Custom

Compute Type: Basic (General Purpose),

Core Count: 8 Driver Cores,

Time to Live 1 hour

I confirm that I have read the links ...

https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance

https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance#optimizing-sinks SinkTooSlow

KranthiPakala-MSFT 45,982 Reputation points Microsoft Employee

2023-10-30T21:44:12.0233333+00:00

@Kieran Wood Just checking in to see if the below information was helpful. If it answers your query, please do click Accept Answer and Yes for "was this answer helpful", as it might be beneficial to other community members reading this thread. If you have any further query, do let us know.

Thank you

KranthiPakala-MSFT 45,982 Reputation points Microsoft Employee

2023-10-30T21:44:12.0233333+00:00

@Kieran Wood Just checking in to see if the below information was helpful. If it answers your query, please do click Accept Answer and Yes for "was this answer helpful", as it might be beneficial to other community members reading this thread. If you have any further query, do let us know.

Thank you

Answer 1

Even though the source activity executed reasonably fast, it's essential to understand the complexity of the CDC.

In many implementations, CDC can introduce latency, especially when handling large datasets.

Sometimes, the source might have complex joins, aggregations, or other operations that slow down the sink.

If your source and sink are in different regions, I assume you can experience added latency. Maybe in this case the source and the sink need to be in the same region if possible. Also, check that other processes aren't competing for resources

Within my Sink of my Data Flow it took 3 minutes 16 seconds to write 106 rows. How do I reduce this time to write rows to a reasonable level?

0 additional answers

Activity