Unexpected Pipeline Slowness / Increase in Data Flow vCore Consumption

Question

Unexpected Pipeline Slowness / Increase in Data Flow vCore Consumption

Angus White 0

Hi,

We have been running the same pipelines on a weekly basis and have seen consistent results when it comes to run time / vCore consumption / cost. A couple of weeks ago however, we noticed that our pipelines were running extremely slowly, and as a result we have consumed a huge amount of V-core processing + our Azure cost has more than doubled.

All our pipelines are impacted, but here is a single example:

Pipeline duration:

07/14/2024 (regular):
30s
07/20/2024 (irregular):
10m 6s

User's image

Dataflow vCore Consumption:
07/14/2024 (regular):
0.0722
07/20/2024 (irregular):
2.614

User's image

Any ideas on what is causing the impact to our pipeline performance? Nothing has changed from a volume perspective with our data volumes / flows. We've made no changes to the pipelines. We are still processing the same number of records.

One theory is that our Azure resource / VM associated with the run time was low on space, would this have an impact?

Thanks,

Angus

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-08-09T18:49:25.5166667+00:00

Hello Angus White,

I am checking to see if you have any further questions here.
Angus White 0 Reputation points

2024-08-12T21:46:44.8866667+00:00

Hi @Bhargava-MSFT ,
Thank you so much for your help, it's much appreciated.

Yes I do have one follow up question. Is there a way to check what Azure resource / VM is associated with a Data Factory instance / pipeline run?

I did look through the Azure Monitor logs and metrics, but nothing has jumped out as being a cause.

Thanks,
Angus
Angus White 0 Reputation points

2024-08-13T16:01:21.1466667+00:00

Hi @Bhargava-MSFT , Thank you so much for your help, it's much appreciated.

Yes I do have one follow up question. Is there a way to check what Azure resource / VM is associated with a Data Factory instance / pipeline run?

I did look through the Azure Monitor logs and metrics, but nothing has jumped out as being a cause.

Thanks, Angus
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-08-13T18:04:51.27+00:00

Hello Angus White,

In ADF, the compute resources used to run pipelines and activities are managed by Integration Runtimes. You need to determine which IR your pipeline is using.

If you are using the Auto Integration Runtime, it's managed by Azure, and you won't have direct access to the underlying compute. If you're using a Self-hosted Integration Runtime, then the resources are within your control.

for SHIR, by logging into the VM and checking CPU, memory, disk space, and network utilization to see if there's any resource contention

Monitor Performance Metrics:

You can monitor the Azure IR performance metrics from the ADF monitoring section as shown below

If you have reviewed the Azure Monitor logs and metrics and have not found any issues, it is possible that the performance degradation is related to resource usage or network issues. You may want to consider scaling up the IR or checking the network connectivity between your ADF workspace and the data source or destination.

How to Scale up:

You can choose a different compute size for your Dataflow runtime in Azure Data Factory. This involves selecting from options like Small, Medium, Large, or even Custom, depending on your file size and processing needs.

Keep in mind that scaling out the nodes will increase the cost of the Azure IR.

I hope this helps.

1 answer

Your answer

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-08-09T18:49:25.5166667+00:00

Hello Angus White,

I am checking to see if you have any further questions here.
Angus White 0 Reputation points

2024-08-12T21:46:44.8866667+00:00

Hi @Bhargava-MSFT ,
Thank you so much for your help, it's much appreciated.

Yes I do have one follow up question. Is there a way to check what Azure resource / VM is associated with a Data Factory instance / pipeline run?

I did look through the Azure Monitor logs and metrics, but nothing has jumped out as being a cause.

Thanks,
Angus
Angus White 0 Reputation points

2024-08-13T16:01:21.1466667+00:00

Hi @Bhargava-MSFT , Thank you so much for your help, it's much appreciated.

Yes I do have one follow up question. Is there a way to check what Azure resource / VM is associated with a Data Factory instance / pipeline run?

I did look through the Azure Monitor logs and metrics, but nothing has jumped out as being a cause.

Thanks, Angus
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-08-13T18:04:51.27+00:00

Hello Angus White,

In ADF, the compute resources used to run pipelines and activities are managed by Integration Runtimes. You need to determine which IR your pipeline is using.

If you are using the Auto Integration Runtime, it's managed by Azure, and you won't have direct access to the underlying compute. If you're using a Self-hosted Integration Runtime, then the resources are within your control.

for SHIR, by logging into the VM and checking CPU, memory, disk space, and network utilization to see if there's any resource contention

Monitor Performance Metrics:

You can monitor the Azure IR performance metrics from the ADF monitoring section as shown below

If you have reviewed the Azure Monitor logs and metrics and have not found any issues, it is possible that the performance degradation is related to resource usage or network issues. You may want to consider scaling up the IR or checking the network connectivity between your ADF workspace and the data source or destination.

How to Scale up:

You can choose a different compute size for your Dataflow runtime in Azure Data Factory. This involves selecting from options like Small, Medium, Large, or even Custom, depending on your file size and processing needs.

Keep in mind that scaling out the nodes will increase the cost of the Azure IR.

I hope this helps.

Answer 1

Hello Angus White,

Welcome to the Microsoft Q&A forum.

It seems like you're experiencing a performance issue with your pipelines, which has led to increased vCore consumption and higher Azure costs. Given that there have been no changes to your data volumes or pipelines, and you're processing the same number of records, there are a few potential factors that could be causing this issue:

If your Azure resource or VM associated with the runtime was low on resources, it could indeed impact performance.
There might be issues with the underlying infrastructure that your pipelines are running on. This could include network latency, hardware failures, or other resource contention issues.
If your data is stored in Azure Data Lake Storage, Azure SQL Database, or other cloud storage, check for any performance issues or I/O bottlenecks during the impacted period.
Check the network connectivity. If there are network connectivity issues between your Azure Synapse Analytics workspace and the data source or destination, it could be causing the pipeline to run slower.

I recommend reviewing Azure Monitor logs and metrics around the time of the slowdown to see if there are any additional details or patterns that could explain the performance degradation. This might help identify whether the issue is related to resource usage, network issues, or something else.

Share via

Unexpected Pipeline Slowness / Increase in Data Flow vCore Consumption

1 answer

Your answer