How to debug an ADF pipeline runnning indefinitely?

Samyak Jain 41

I followed the official tutorial on how to run a python script using ADF: https://learn.microsoft.com/en-us/azure/batch/tutorial-run-python-batch-azure-data-factory but the pipeline is running indefinitely. I am unable to find the cause of this issue.
I need help with the same.

Accepted answer

ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee

2021-12-13T13:56:37.35+00:00
Hi @Samyak Jain ,

Thank you for posting query in Microsoft Q&A Platform.

Reason behind running pipeline indefinitely is due to timeout value set inside Custom activity. By default custom activity will have timeout set as 7 days.

ADF may not getting response back from Batch services and hence we may be ending up seeing pipeline running for long time.

Could you please try below options and see if that helps.

Make sure your Azure Batch linked service test connection is successful.

Monitor your Azure Batch log files. Click here to know more about it.

Hope this will helps. Please let us know if any further queries. Thank you.

-------------

Please consider hitting Accept Answer button. Accepted answers helps community as well.
Please sign in to rate this answer.
ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee

2021-12-15T14:35:52.767+00:00

Hi @Samyak Jain ,

Just checking in to see if the below answer helped. If this answers your query, do click and upvote for the same. And, if you have any further query do let us know.

Samyak Jain 41 Reputation points

2021-12-16T07:26:27.513+00:00

Hi @ShaikMaheer-MSFT ,

I tried the solutions you provided,

I checked the connections(with storage and batch services) while creating the pipeline and also tested them before running the pipeline. The connections were successful.

Checked the logs from the batch service but couldn't see any log getting created, just that the job was running(which ran for a lot of time before I stopped it)

Is there any other resolution? Does the fact that the account is a free one currently make a difference to the resolution?

ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee

2021-12-21T16:54:37.967+00:00

Hi @Samyak Jain ,

Free subscription will have some limitations for few services. If I am not wrong ADF & Azure batch services don't have any restrictions as such. Did you tried setting your timeout values less may be to few mins in custom activity? Also, you tried to check logs se explained in below image? Feel free to share those log details if it is okay for you.
Sign in to comment

1 additional answer

Samyak Jain 41 Reputation points

2021-12-22T07:18:18.03+00:00

Hi @ShaikMaheer-MSFT ,

I think I found the issue here. I am following the tutorial https://learn.microsoft.com/en-us/azure/batch/tutorial-run-python-batch-azure-data-factory with a free account. Upon checking the quota, I found out that FSv2 VM's(as suggested to be used in the tutorial) aren't free to use for free accounts.
I feel that was what caused the job to run indefinitely as when I tried the same with a simpler machine, I was able to run the job.
Please let me know your views on the same.
Please sign in to rate this answer.
ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee

2021-12-23T16:34:49.02+00:00

Hi @Samyak Jain ,

That's great catch. Thank you for sharing details. Could you please consider marking your answer as Accepted Answer? Accepted answers helps community as well.
Sign in to comment