Azure Machine Learning
An Azure machine learning service for building and deploying models.
3,268 questions
We are getting frequent timeouts on mlflow on machine learning
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 66%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Stéphane Renou
0
Reputation points
On frequent basis, our code is getting timed out after 2 minutes when trying to get status of a ML flow job.
A recent example is showing the response as:
Retrying (Retry(total=6, connect=7, read=6, redirect=7, status=7)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='francecentral.api.azureml.ms', port=443): Read timed out. (read timeout=120)")': /mlflow/v2.0/subscriptions/28377576-89a8-4e51-8209-45265b06caee/resourceGroups/rg-prod-sales-azure/providers/Microsoft.MachineLearningServices/workspaces/mlw-sales-azure-prod-lc/api/2.0/mlflow/runs/search
we are making the call from the compute:
| /subscriptions/28377576-89a8-4e51-8209-45265b06caee/resourceGroups/RG-PROD-MAIN/providers/Microsoft.Compute/virtualMachines/LearnVM | |
|---|---|
| /subscriptions/28377576-89a8-4e51-8209-45265b06caee/resourceGroups/RG-PROD-MAIN/providers/Microsoft.Compute/virtualMachines/LearnVM |
The call fails at:
/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py
while doing: urlopen
We are using mlflow library in python and we don't understand why on frequent basis, those calls are timing out after 2 minutes while all our infrastructure is located next to each other.
We are using libraries:
azure-ai-ml version 1.26.0
azureml-mlflow version 1.59.0.post1
mlflow-skinny version 2.21.1
Azure Machine Learning
{count} votes
-
Azar 28,310 Reputation points MVP Moderator2025-04-23T16:32:51.5233333+00:00 Hi there Stéphane Renou
Thanks for using QandA platform
Even though your resources are in the same region, the
mlflow/runs/searchendpoint can become slow, especially if you're querying a large number of runs or using complex filters. The 2-minute timeout usually means the backend didn't respond in time, not necessarily a network issue on your side. Also, usingmlflow-skinnywith older versions likeazureml-mlflow 1.59.0.post1may cause compatibility issues or unexpected delays.So try reducing the size of your queries, upgrading to the latest versions of all related libraries, and consider switching from
mlflow-skinnyto the fullmlflowpackage if possible.If this continues, it’s worth raising a support ticket with Azure
https://mlflow.org/docs/latest/rest-api.html#search-runs
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-errors
If this helps kindly accept the answer thanks much.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 15%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Manas Mohanty 3,955 Reputation points Microsoft External Staff Moderator2025-04-23T16:44:45.0966667+00:00 Agreed to Azar' pointer on using full "mlflow" package
It seems you are facing timeout while doing urlopen. Issue might be due to long page loading time or URL is not accessible inside your secured network. Please try with URLs from your blob storage.
You can increase timeout and no of retries from deployment if loading time is high.
You can add try catch block in urlopen block to find exact issue. Please load the model or image locally before testing in a job or online endpoint to find the issue.
from socket import timeout from urllib.error import HTTPError, URLError try: response = urllib.request.urlopen(url, timeout=10).read().decode('utf-8') except HTTPError as error: logging.error('HTTP Error: Data not retrieved because %s\nURL: %s', error, url) except URLError as error: if isinstance(error.reason, timeout): logging.error('Timeout Error: Data not retrieved because %s\nURL: %s', error, url) else: logging.error('URL Error: Data not retrieved because %s\nURL: %s', error, url) else: logging.info('Access successful.')Please share the full log in private chat for further assistance.
Attached relevant troubleshooting guide for reference -
Troubleshoot online endpoint deployment - Azure Machine Learning | Microsoft Learn
Hope it helps.
Thank you.
-
Vahid Ghafarpour 23,280 Reputation points Moderator2025-04-23T16:47:38.73+00:00 It can be as a result of complex query or mass result.
Maybe adding pagination can help you
-
Stéphane Renou 0 Reputation points
2025-04-24T08:49:27.8766667+00:00 Hello, thanks a lot for all those great feedbacks.
We are using mlflow skinny as we are only tracking the progress of our operations, so we don't need the full hammer of mlflow for just that tracking and we can reduce the size of our librairies that we ship.Concerning the versions being used, those are just the version previous to the current latest, but I would try to upgrade our production env with the latest to see if by any chance the bug has disappeared.
Concerning the filters we are doing very basic queries to get the status of a single job without any other filter, so I am not sure the complexity of the query could explain the timeout to get the result.
I guess I would trigger a ticket with azure to see what is going on.
-
Manas Mohanty 3,955 Reputation points Microsoft External Staff Moderator
2025-04-24T09:09:34.1066667+00:00 Wanted to know if it was a model or file in urlopen operation.
If models or files are large ones, it will take some time to load and time out if timeout and retry param is not increase.
You can load the models from blob containers instead if the workspace is virtual network protected one.
Loading those files locally in notebook can help you debug the cause properly.
Thank you.
-
Manas Mohanty 3,955 Reputation points Microsoft External Staff Moderator
2025-04-25T15:52:01.32+00:00 We are still waiting to hear from you. Please let us know if you had further queries on it.
Thank you.
-
Manas Mohanty 3,955 Reputation points Microsoft External Staff Moderator
2025-04-28T06:32:23.9066667+00:00 We have not heard from you. Hope the pointers shared were useful to you.
Thank you.
-
Stéphane Renou 0 Reputation points
2025-04-28T08:07:49.2833333+00:00 Hello,
Sorry for not answering. The urlopen refers to a query to the mlflow server that is running a job to know the status of the job.
-
Manas Mohanty 3,955 Reputation points Microsoft External Staff Moderator
2025-04-28T11:13:03.5766667+00:00 I have requested details in private chat for further assistance. But it seems that VM might not be allowing connection to view status of pipe run, you might need to add that VM as contributor to your AML resource if not done already.
If it is virtual network protected, you need to add inbound connection from your VM to your AML workspace subnet.
Thank you
-
Stéphane Renou 0 Reputation points
2025-04-28T12:32:07.22+00:00 Hello,
The issue we are seeing is occuring on some cases, while most of the time all is ok, so I don't think any configuration issue could be the root cause, else if would always fail?
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 77%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sampath 2,845 Reputation points Microsoft External Staff Moderator2025-05-01T16:06:37.5133333+00:00 Hello @ Stéphane Renou, requesting you check your private message and reply there. Thank you.
Sign in to comment
Sign in to answer