We are getting frequent timeouts on mlflow on machine learning

Question

We are getting frequent timeouts on mlflow on machine learning

Stéphane Renou 20

On frequent basis, our code is getting timed out after 2 minutes when trying to get status of a ML flow job.

A recent example is showing the response as:

Retrying (Retry(total=6, connect=7, read=6, redirect=7, status=7)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='francecentral.api.azureml.ms', port=443): Read timed out. (read timeout=120)")': /mlflow/v2.0/subscriptions/28377576-89a8-4e51-8209-45265b06caee/resourceGroups/rg-prod-sales-azure/providers/Microsoft.MachineLearningServices/workspaces/mlw-sales-azure-prod-lc/api/2.0/mlflow/runs/search

we are making the call from the compute:

	/subscriptions/28377576-89a8-4e51-8209-45265b06caee/resourceGroups/RG-PROD-MAIN/providers/Microsoft.Compute/virtualMachines/LearnVM
	/subscriptions/28377576-89a8-4e51-8209-45265b06caee/resourceGroups/RG-PROD-MAIN/providers/Microsoft.Compute/virtualMachines/LearnVM

The call fails at:

/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py

while doing: urlopen

We are using mlflow library in python and we don't understand why on frequent basis, those calls are timing out after 2 minutes while all our infrastructure is located next to each other.

We are using libraries:
azure-ai-ml version 1.26.0

azureml-mlflow version 1.59.0.post1
mlflow-skinny version 2.21.1

Azar 31,725 Reputation points MVP Volunteer Moderator

2025-04-23T16:32:51.5233333+00:00

Hi there Stéphane Renou

Thanks for using QandA platform

Even though your resources are in the same region, the mlflow/runs/search endpoint can become slow, especially if you're querying a large number of runs or using complex filters. The 2-minute timeout usually means the backend didn't respond in time, not necessarily a network issue on your side. Also, using mlflow-skinny with older versions like azureml-mlflow 1.59.0.post1 may cause compatibility issues or unexpected delays.

So try reducing the size of your queries, upgrading to the latest versions of all related libraries, and consider switching from mlflow-skinny to the full mlflow package if possible.

If this continues, it’s worth raising a support ticket with Azure

https://mlflow.org/docs/latest/rest-api.html#search-runs

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-errors

If this helps kindly accept the answer thanks much.
Manas R Mohanty 17,270 Reputation points Moderator

2025-04-23T16:44:45.0966667+00:00
Hi Stéphane Renou

Agreed to Azar' pointer on using full "mlflow" package

It seems you are facing timeout while doing urlopen. Issue might be due to long page loading time or URL is not accessible inside your secured network. Please try with URLs from your blob storage.

You can increase timeout and no of retries from deployment if loading time is high.

You can add try catch block in urlopen block to find exact issue. Please load the model or image locally before testing in a job or online endpoint to find the issue.

from socket import timeout from urllib.error import HTTPError, URLError try: response = urllib.request.urlopen(url, timeout=10).read().decode('utf-8') except HTTPError as error: logging.error('HTTP Error: Data not retrieved because %s\nURL: %s', error, url) except URLError as error: if isinstance(error.reason, timeout): logging.error('Timeout Error: Data not retrieved because %s\nURL: %s', error, url) else: logging.error('URL Error: Data not retrieved because %s\nURL: %s', error, url) else: logging.info('Access successful.')

Please share the full log in private chat for further assistance.

Attached relevant troubleshooting guide for reference -

Troubleshoot online endpoint deployment - Azure Machine Learning | Microsoft Learn

Relevant thread

Hope it helps.

Thank you.
Vahid Ghafarpour 23,605 Reputation points

2025-04-23T16:47:38.73+00:00

It can be as a result of complex query or mass result.

Maybe adding pagination can help you
Stéphane Renou 20 Reputation points

2025-04-24T08:49:27.8766667+00:00

Hello, thanks a lot for all those great feedbacks.
We are using mlflow skinny as we are only tracking the progress of our operations, so we don't need the full hammer of mlflow for just that tracking and we can reduce the size of our librairies that we ship.

Concerning the versions being used, those are just the version previous to the current latest, but I would try to upgrade our production env with the latest to see if by any chance the bug has disappeared.

Concerning the filters we are doing very basic queries to get the status of a single job without any other filter, so I am not sure the complexity of the query could explain the timeout to get the result.

I guess I would trigger a ticket with azure to see what is going on.
Manas R Mohanty 17,270 Reputation points Moderator

2025-04-24T09:09:34.1066667+00:00

Hi Stéphane Renou

Wanted to know if it was a model or file in urlopen operation.

If models or files are large ones, it will take some time to load and time out if timeout and retry param is not increase.

You can load the models from blob containers instead if the workspace is virtual network protected one.

Loading those files locally in notebook can help you debug the cause properly.

Thank you.
Manas R Mohanty 17,270 Reputation points Moderator

2025-04-25T15:52:01.32+00:00

Hi Stéphane Renou

We are still waiting to hear from you. Please let us know if you had further queries on it.

Thank you.
Manas R Mohanty 17,270 Reputation points Moderator

2025-04-28T06:32:23.9066667+00:00

Hi Stéphane Renou

We have not heard from you. Hope the pointers shared were useful to you.

Thank you.
Stéphane Renou 20 Reputation points

2025-04-28T08:07:49.2833333+00:00

Hello,

Sorry for not answering. The urlopen refers to a query to the mlflow server that is running a job to know the status of the job.
Manas R Mohanty 17,270 Reputation points Moderator

2025-04-28T11:13:03.5766667+00:00

Hi Stéphane Renou

I have requested details in private chat for further assistance. But it seems that VM might not be allowing connection to view status of pipe run, you might need to add that VM as contributor to your AML resource if not done already.

If it is virtual network protected, you need to add inbound connection from your VM to your AML workspace subnet.

Reference - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-access-azureml-behind-firewall?view=azureml-api-2&tabs=ipaddress%2Cpublic

Thank you
Stéphane Renou 20 Reputation points

2025-04-28T12:32:07.22+00:00

Hello,

The issue we are seeing is occuring on some cases, while most of the time all is ok, so I don't think any configuration issue could be the root cause, else if would always fail?
Sampath 3,885 Reputation points Microsoft External Staff Moderator

2025-05-01T16:06:37.5133333+00:00

Hello @ Stéphane Renou, requesting you check your private message and reply there. Thank you.
Stéphane Renou 20 Reputation points

2025-07-25T08:54:33.9866667+00:00

Unfortunately the issue came back and is hitting us on frequent basis. I felt that upgrading mlflow solved the issue, it just hide it for a few days and now the issue is back. I was contacted by a Microsoft Engineer at some point, can someone contact me again to help on this?

1 answer

Your answer

Azar 31,725 Reputation points MVP Volunteer Moderator

2025-04-23T16:32:51.5233333+00:00

Hi there Stéphane Renou

Thanks for using QandA platform

Even though your resources are in the same region, the mlflow/runs/search endpoint can become slow, especially if you're querying a large number of runs or using complex filters. The 2-minute timeout usually means the backend didn't respond in time, not necessarily a network issue on your side. Also, using mlflow-skinny with older versions like azureml-mlflow 1.59.0.post1 may cause compatibility issues or unexpected delays.

So try reducing the size of your queries, upgrading to the latest versions of all related libraries, and consider switching from mlflow-skinny to the full mlflow package if possible.

If this continues, it’s worth raising a support ticket with Azure

https://mlflow.org/docs/latest/rest-api.html#search-runs

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-errors

If this helps kindly accept the answer thanks much.
Manas R Mohanty 17,270 Reputation points Moderator

2025-04-23T16:44:45.0966667+00:00

Hi Stéphane Renou

Agreed to Azar' pointer on using full "mlflow" package

It seems you are facing timeout while doing urlopen. Issue might be due to long page loading time or URL is not accessible inside your secured network. Please try with URLs from your blob storage.

You can increase timeout and no of retries from deployment if loading time is high.

You can add try catch block in urlopen block to find exact issue. Please load the model or image locally before testing in a job or online endpoint to find the issue.

from socket import timeout from urllib.error import HTTPError, URLError try: response = urllib.request.urlopen(url, timeout=10).read().decode('utf-8') except HTTPError as error: logging.error('HTTP Error: Data not retrieved because %s\nURL: %s', error, url) except URLError as error: if isinstance(error.reason, timeout): logging.error('Timeout Error: Data not retrieved because %s\nURL: %s', error, url) else: logging.error('URL Error: Data not retrieved because %s\nURL: %s', error, url) else: logging.info('Access successful.')

Please share the full log in private chat for further assistance.

Attached relevant troubleshooting guide for reference -

Troubleshoot online endpoint deployment - Azure Machine Learning | Microsoft Learn

Relevant thread

Hope it helps.

Thank you.
Vahid Ghafarpour 23,605 Reputation points

2025-04-23T16:47:38.73+00:00

It can be as a result of complex query or mass result.

Maybe adding pagination can help you
Stéphane Renou 20 Reputation points

2025-04-24T08:49:27.8766667+00:00

Hello, thanks a lot for all those great feedbacks.
We are using mlflow skinny as we are only tracking the progress of our operations, so we don't need the full hammer of mlflow for just that tracking and we can reduce the size of our librairies that we ship.

Concerning the versions being used, those are just the version previous to the current latest, but I would try to upgrade our production env with the latest to see if by any chance the bug has disappeared.

Concerning the filters we are doing very basic queries to get the status of a single job without any other filter, so I am not sure the complexity of the query could explain the timeout to get the result.

I guess I would trigger a ticket with azure to see what is going on.
Manas R Mohanty 17,270 Reputation points Moderator

2025-04-24T09:09:34.1066667+00:00

Hi Stéphane Renou

Wanted to know if it was a model or file in urlopen operation.

If models or files are large ones, it will take some time to load and time out if timeout and retry param is not increase.

You can load the models from blob containers instead if the workspace is virtual network protected one.

Loading those files locally in notebook can help you debug the cause properly.

Thank you.
Manas R Mohanty 17,270 Reputation points Moderator

2025-04-25T15:52:01.32+00:00

Hi Stéphane Renou

We are still waiting to hear from you. Please let us know if you had further queries on it.

Thank you.
Manas R Mohanty 17,270 Reputation points Moderator

2025-04-28T06:32:23.9066667+00:00

Hi Stéphane Renou

We have not heard from you. Hope the pointers shared were useful to you.

Thank you.
Stéphane Renou 20 Reputation points

2025-04-28T08:07:49.2833333+00:00

Hello,

Sorry for not answering. The urlopen refers to a query to the mlflow server that is running a job to know the status of the job.
Manas R Mohanty 17,270 Reputation points Moderator

2025-04-28T11:13:03.5766667+00:00

Hi Stéphane Renou

I have requested details in private chat for further assistance. But it seems that VM might not be allowing connection to view status of pipe run, you might need to add that VM as contributor to your AML resource if not done already.

If it is virtual network protected, you need to add inbound connection from your VM to your AML workspace subnet.

Reference - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-access-azureml-behind-firewall?view=azureml-api-2&tabs=ipaddress%2Cpublic

Thank you
Stéphane Renou 20 Reputation points

2025-04-28T12:32:07.22+00:00

Hello,

The issue we are seeing is occuring on some cases, while most of the time all is ok, so I don't think any configuration issue could be the root cause, else if would always fail?
Sampath 3,885 Reputation points Microsoft External Staff Moderator

2025-05-01T16:06:37.5133333+00:00

Hello @ Stéphane Renou, requesting you check your private message and reply there. Thank you.
Stéphane Renou 20 Reputation points

2025-07-25T08:54:33.9866667+00:00

Unfortunately the issue came back and is hitting us on frequent basis. I felt that upgrading mlflow solved the issue, it just hide it for a few days and now the issue is back. I was contacted by a Microsoft Engineer at some point, can someone contact me again to help on this?

Answer 1

Mohammad Bani Hani 6 Microsoft Employee

This issue resolved on this item. https://github.com/mlflow/mlflow/issues/21470

The latest mlflow should have this resolved, however, it requires sometimes to be released under https://pypi.org/project/azureml-mlflow/#description

0 comments