Azure Machine Learning Python SDK Error "RunIDConflict"

Question

Azure Machine Learning Python SDK Error "RunIDConflict"

Nils 1

Hello everybody,

I am currently learning Azure Machine Learning using the learning paths and labs. The lab scripts give me the following error I could not find anywhere yet.

"Run IDs must be unique within a workspace and can only be used once. Ensure multiple runs with the same ID are not submitted simultaneously."

I am running the scripts locally. I have tried both Jupyter Notebook and VS Code. The error occurs exactly after 65 seconds when running experiment scripts. Both lab 1 with script config and lab 2 with an estimator give me the error. Running experiments directly within the IPython script works and I can see the results in the Azure web GUI. My SDK version is 1.21. When I did this in May 2020 with SDK version 1.15 I did not receive this error.

The respective code blocks that throw the errors copied from the lab scripts:

Lab 1:

import os
import sys
from azureml.core import Experiment, ScriptRunConfig
from azureml.widgets import RunDetails


# Create a script config
script_config = ScriptRunConfig(source_directory=experiment_folder, 
                      script='diabetes_experiment.py') 

# submit the experiment
experiment = Experiment(workspace = ws, name = 'diabetes-experiment')
run = experiment.submit(config=script_config)
RunDetails(run).show()
run.wait_for_completion()

Lab 2:

from azureml.train.estimator import Estimator
from azureml.core import Experiment

# Create an estimator
estimator = Estimator(source_directory=training_folder,
                      entry_script='diabetes_training.py',
                      compute_target='local',
                      conda_packages=['scikit-learn']
                      )

# Create an experiment
experiment_name = 'diabetes-training'
experiment = Experiment(workspace = ws, name = experiment_name)

# Run the experiment based on the estimator
run = experiment.submit(config=estimator)
run.wait_for_completion(show_output=True)

Can you tell me how to fix this error? Thanks in advance.

Regards

GiftA-MSFT 11,181 Reputation points Moderator

2021-02-04T22:15:48.373+00:00

Hi, we are reviewing this issue and will get back to you soon. Thanks.
Rohit Mungi 49,126 Reputation points Microsoft Employee Moderator

2021-02-08T15:50:26.807+00:00

@Nils Could you please try to use a different experiment name in both the cases?
Nils 1 Reputation point

2021-02-09T14:26:22.423+00:00

@Rohit Mungi Thanks for your answer. I re-tried with simpler and new experiment names "experiment1", "experiment2", and "experiment3". The behavior is the same as before. The direct run version works, the other two calling the sub-scripts fail with the same error after 65 seconds.
Rohit Mungi 49,126 Reputation points Microsoft Employee Moderator

2021-02-09T14:38:44.293+00:00

@Nils Just curious to check if the diabetes_experiment.py file is created in the folder in the cell above the code you mentioned?

This image is from Lab1 link you posted.

The timeout you mentioned seems to indicate the config script is not available and your experiment might be timing out. It might be easier to check your experiments directly on the Azure ml portal ml.azure.com to check for more details in the logs of your experiment or runs.
Nils 1 Reputation point

2021-02-10T08:19:00.163+00:00
@Rohit Mungi Thanks for your reply. It is indeed created and if I run the following code

%run diabetes-experiment-files/diabetes_experiment.py

I receive the following output

Attempted to log scalar metric observations: 10000 Analyzing 10000 rows of data 0 6656 1 3344 Name: Diabetic, dtype: int64 Attempted to log scalar metric Label:0: 6656 Attempted to log scalar metric Label:1: 3344

This seems reasonable to me as there is no run context to log the values but shows that the script in itself is functional.

I currently dont have access to compute instances to test it directly in the Azure ML Portal.

If it helps, this is the error traceback (exceeds character limit as code block, hence as image):
Rohit Mungi 49,126 Reputation points Microsoft Employee Moderator

2021-02-10T15:50:06.493+00:00

@Nils Is this showing the same error as you have initially seen? The above screen shot shows the traceback but I couldn't check if this is the same error message.
Also, on the portal it should show the experiments on the tab on left hand side but in this case the submit is failing so I think it would not help us much.
Nils 1 Reputation point

2021-02-11T09:39:43.13+00:00
@Rohit Mungi It is the same error, yes. I can see the experiments and each failed run in the portal, but the only error message there is

Service Error

Received 500 from a service request

2 answers

Your answer

GiftA-MSFT 11,181 Reputation points Moderator

2021-02-04T22:15:48.373+00:00

Hi, we are reviewing this issue and will get back to you soon. Thanks.
Rohit Mungi 49,126 Reputation points Microsoft Employee Moderator

2021-02-08T15:50:26.807+00:00

@Nils Could you please try to use a different experiment name in both the cases?
Nils 1 Reputation point

2021-02-09T14:26:22.423+00:00

@Rohit Mungi Thanks for your answer. I re-tried with simpler and new experiment names "experiment1", "experiment2", and "experiment3". The behavior is the same as before. The direct run version works, the other two calling the sub-scripts fail with the same error after 65 seconds.
Rohit Mungi 49,126 Reputation points Microsoft Employee Moderator

2021-02-09T14:38:44.293+00:00

@Nils Just curious to check if the diabetes_experiment.py file is created in the folder in the cell above the code you mentioned?

This image is from Lab1 link you posted.

The timeout you mentioned seems to indicate the config script is not available and your experiment might be timing out. It might be easier to check your experiments directly on the Azure ml portal ml.azure.com to check for more details in the logs of your experiment or runs.
Nils 1 Reputation point

2021-02-10T08:19:00.163+00:00

@Rohit Mungi Thanks for your reply. It is indeed created and if I run the following code

%run diabetes-experiment-files/diabetes_experiment.py

I receive the following output

Attempted to log scalar metric observations: 10000 Analyzing 10000 rows of data 0 6656 1 3344 Name: Diabetic, dtype: int64 Attempted to log scalar metric Label:0: 6656 Attempted to log scalar metric Label:1: 3344

This seems reasonable to me as there is no run context to log the values but shows that the script in itself is functional.

I currently dont have access to compute instances to test it directly in the Azure ML Portal.

If it helps, this is the error traceback (exceeds character limit as code block, hence as image):
Rohit Mungi 49,126 Reputation points Microsoft Employee Moderator

2021-02-10T15:50:06.493+00:00

@Nils Is this showing the same error as you have initially seen? The above screen shot shows the traceback but I couldn't check if this is the same error message.
Also, on the portal it should show the experiments on the tab on left hand side but in this case the submit is failing so I think it would not help us much.
Nils 1 Reputation point

2021-02-11T09:39:43.13+00:00

@Rohit Mungi It is the same error, yes. I can see the experiments and each failed run in the portal, but the only error message there is

Service Error

Received 500 from a service request

Answer 1

Anyone have find a solution to it? I meet the same issue.

  1 src.run_config.environment = system_managed_env

----> 2 run = exp.submit(src)

C:\Anaconda\lib\site-packages\azureml\core\experiment.py in submit(self, config, tags, **kwargs)
218 submit_func = get_experiment_submit(config)
219 with self._log_context("submit config {}".format(config.class.name)):
--> 220 run = submit_func(config, self.workspace, self.name, **kwargs)
221 if tags is not None:
222 run.set_tags(tags)

C:\Anaconda\lib\site-packages\azureml\core\script_run_config.py in submit(script_run_config, workspace, experiment_name, run_id, _parent_run_id, credential_passthrough)
62 run = _commands.start_run(project, run_config,
63 telemetry_values=script_run_config._telemetry_values,
---> 64 run_id=run_id, parent_run_id=_parent_run_id)
65 run.add_properties(global_tracking_info_registry.gather_all(script_run_config.source_directory))
66

C:\Anaconda\lib\site-packages\azureml_execution_commands.py in start_run(project_object, run_config_object, run_id, injected_files, telemetry_values, parent_run_id, prepare_only, check)
117 raise ExperimentExecutionException("Can not check preparation of local targets")
118 return _start_internal_local_cloud(project_object, run_config_object,
--> 119 **shared_start_run_kwargs)
120 else:
121 return _start_internal(project_object, run_config_object, prepare_check=check,

C:\Anaconda\lib\site-packages\azureml_execution_commands.py in _start_internal_local_cloud(project_object, run_config_object, prepare_only, custom_target_dict, run_id, injected_files, telemetry_values, parent_run_id)
267
268 response = ClientBase._execute_func(requests.post, uri, files=files, headers=headers)
--> 269 _raise_request_error(response, "starting run")
270
271 invocation_zip_path = os.path.join(project_temp_dir, "invocation.zip")

C:\Anaconda\lib\site-packages\azureml_execution_commands.py in _raise_request_error(response, action)
569 # response.text is a JSON from execution service.
570 response_message = get_http_exception_response_string(response)
--> 571 raise ExperimentExecutionException(response_message)
572
573

ExperimentExecutionException: ExperimentExecutionException:
Message: {
"error_details": {
"componentName": "execution",
"correlation": {
"operation": "a6adb9078e64c642a0828802f831f801",
"request": "aea5e681ddf5e549"
},
"environment": "westeurope",
"error": {
"code": "UserError",
"innerError": {
"code": "Conflict",
"innerError": {
"code": "RunIDConflict"
}
},
"message": "Run IDs must be unique within a workspace and can only be used once. Ensure multiple runs with the same ID are not submitted simultaneously.",
"messageFormat": "Run IDs must be unique within a workspace and can only be used once. Ensure multiple runs with the same ID are not submitted simultaneously."
},

Answer 2

GiftA-MSFT 11,181 Moderator

Please try to run the labs using Azure ML Studio (Notebooks), I was able to run it there without any errors. However, while running locally, I ran into a different error. Hence, the error may be related to your local environment. I also noticed the Labs have been archived, so please report the issue directly to Microsoft Learn team. If you're trying to explore Azure ML, please follow the tutorials in our official documentation. Hope this helps!

0 comments

Share via

Azure Machine Learning Python SDK Error "RunIDConflict"

2 answers

Your answer