Data Pipeline Got Struck in between and could not move forward.

Question

Data Pipeline Got Struck in between and could not move forward.

Rekulapalli, Karthikeya 0

Hi ,

I am Running a short pipeline just has couple of transformations and later uploads data into Snowflake, my pipeline got struck in between. but later after rerunning it after cancelation it could not acquire resources.

User's image

Rekulapalli, Karthikeya 0 Reputation points

2023-12-12T20:17:13.3366667+00:00

Hashing used is round robin 25 partitions. Data Flow cluster size medium??
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2023-12-13T14:33:12.62+00:00
@Rekulapalli, Karthikeya

Thanks for recahing out MS Q&A

Check the status of the resources: Make sure that the resources that your pipeline is using (e.g., compute resources, data stores) are available and functioning correctly. You can check the status of these resources in the Azure portal.

Check the logs: Check the logs for your pipeline to see if there are any error messages or other issues that could be causing the pipeline to fail. You can view the logs for your pipeline in the Azure portal.

Increase the resources: If your pipeline is failing to acquire resources, you may need to increase the resources that are available to your pipeline. For example, you may need to increase the number of compute nodes or increase the size of the compute nodes.

Check the dependencies: Make sure that any dependencies that your pipeline has (e.g., linked services, datasets) are configured correctly and are available. If any of these dependencies are missing or misconfigured, your pipeline may fail to acquire resources.

Retry the pipeline: If none of these solutions work, you may need to retry your pipeline at a later time. Sometimes, resource availability issues can be temporary and may resolve themselves over time.

Hope this helps. Do let us know if you any further queries.
Rekulapalli, Karthikeya 0 Reputation points

2023-12-13T15:04:57.17+00:00

File size is 400 MB and I don't understand Why?
Rekulapalli, Karthikeya 0 Reputation points

2023-12-13T15:13:13.1933333+00:00

Still same error after 7 ours of Pipeline run it result of in failed and Error 149 Internal Server Error:Spark cluster not found. Please try restarting debug session.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2023-12-14T15:13:35.63+00:00

@Rekulapalli, Karthikeya

Retry Mechanism: You could implement a retry mechanism in your pipeline. This can be done by setting two variables: Pipeline_status (default value: Fail) and Max_loop_count (default value: 0). All activities should be inside an ‘Until’ activity which will have an expression or (equals (Pipeline_status,'Success'),equals (pMax_rerun_count,Max_loop_count). The first activity inside ‘Until’ activity will be ‘Set Variable’ activity that increments the value of variable Max_loop_count by 1. The final activity inside ‘Until’ activity will be to ‘Set variable’ activity that sets Pipeline_status to “Success”. This setup ensures that all intended activities inside the ‘Until’ block run until the intended activities in the pipeline complete successfully<sup>1</sup>.
Rekulapalli, Karthikeya 0 Reputation points

2023-12-14T17:31:56.8733333+00:00

@phemanth that is not working to still acquiring compute
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2023-12-15T06:13:12.8333333+00:00
@Rekulapalli, Karthikeya

Here are a few more things you could try:

Time to Live (TTL): If your pipelines contain multiple sequential data flows, you can enable a TTL value. This keeps a cluster alive for a certain period of time after its execution completes. If a new job starts using the Integration Runtime during the TTL time, it will reuse the existing cluster and startup time will be greatly reduced.

Quick Re-use Option: You can minimize the startup time of warm clusters (already spun up cluster) by setting the Quick re-use option in the Azure Integration Runtime under Data Flow Properties.

If these suggestions don’t resolve the issue, please do let us know.

Your answer

Rekulapalli, Karthikeya 0 Reputation points

2023-12-12T20:17:13.3366667+00:00

Hashing used is round robin 25 partitions. Data Flow cluster size medium??
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2023-12-13T14:33:12.62+00:00

@Rekulapalli, Karthikeya

Thanks for recahing out MS Q&A

Check the status of the resources: Make sure that the resources that your pipeline is using (e.g., compute resources, data stores) are available and functioning correctly. You can check the status of these resources in the Azure portal.

Check the logs: Check the logs for your pipeline to see if there are any error messages or other issues that could be causing the pipeline to fail. You can view the logs for your pipeline in the Azure portal.

Increase the resources: If your pipeline is failing to acquire resources, you may need to increase the resources that are available to your pipeline. For example, you may need to increase the number of compute nodes or increase the size of the compute nodes.

Check the dependencies: Make sure that any dependencies that your pipeline has (e.g., linked services, datasets) are configured correctly and are available. If any of these dependencies are missing or misconfigured, your pipeline may fail to acquire resources.

Retry the pipeline: If none of these solutions work, you may need to retry your pipeline at a later time. Sometimes, resource availability issues can be temporary and may resolve themselves over time.

Hope this helps. Do let us know if you any further queries.
Rekulapalli, Karthikeya 0 Reputation points

2023-12-13T15:04:57.17+00:00

File size is 400 MB and I don't understand Why?
Rekulapalli, Karthikeya 0 Reputation points

2023-12-13T15:13:13.1933333+00:00

Still same error after 7 ours of Pipeline run it result of in failed and Error 149 Internal Server Error:Spark cluster not found. Please try restarting debug session.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2023-12-14T15:13:35.63+00:00

@Rekulapalli, Karthikeya

Retry Mechanism: You could implement a retry mechanism in your pipeline. This can be done by setting two variables: Pipeline_status (default value: Fail) and Max_loop_count (default value: 0). All activities should be inside an ‘Until’ activity which will have an expression or (equals (Pipeline_status,'Success'),equals (pMax_rerun_count,Max_loop_count). The first activity inside ‘Until’ activity will be ‘Set Variable’ activity that increments the value of variable Max_loop_count by 1. The final activity inside ‘Until’ activity will be to ‘Set variable’ activity that sets Pipeline_status to “Success”. This setup ensures that all intended activities inside the ‘Until’ block run until the intended activities in the pipeline complete successfully<sup>1</sup>.
Rekulapalli, Karthikeya 0 Reputation points

2023-12-14T17:31:56.8733333+00:00

@phemanth that is not working to still acquiring compute
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2023-12-15T06:13:12.8333333+00:00

@Rekulapalli, Karthikeya

Here are a few more things you could try:

Time to Live (TTL): If your pipelines contain multiple sequential data flows, you can enable a TTL value. This keeps a cluster alive for a certain period of time after its execution completes. If a new job starts using the Integration Runtime during the TTL time, it will reuse the existing cluster and startup time will be greatly reduced.

Quick Re-use Option: You can minimize the startup time of warm clusters (already spun up cluster) by setting the Quick re-use option in the Azure Integration Runtime under Data Flow Properties.

If these suggestions don’t resolve the issue, please do let us know.

Share via

Data Pipeline Got Struck in between and could not move forward.

Your answer