This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 11%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJA%3C/text%3E%3C/svg%3E)
Hi,
I am trying to create a ML pipeline job using Azure CLI V2 commands. pls find the command below :
az ml job create --file ./mlops/azureml/train/train-pipeline.yml --resource-group myresource_group \
--workspace-name myworkspace --query name -o tsv
I am using Service principal with contributor access on resource group level to authorize the command. But on running the command, it failed to create the ML Pipeline job with following error :
azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'This request is not authorized to perform this operation.'
1016ErrorCode:AuthorizationFailure
1017
1018ERROR: cli.azure.cli.core.azclierror: Operation returned an invalid status 'This request is not authorized to perform this operation.'
1019ErrorCode:AuthorizationFailure
1020ERROR: az_command_data_logger: Operation returned an invalid status 'This request is not authorized to perform this operation.'
1021ErrorCode:AuthorizationFailure
1022DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f6f3d7708b0>]
1023INFO: az_command_data_logger: exit code: 1
1024INFO: cli.main: Command ran in 8.520 seconds (init: 0.336, invoke: 8.184)
1025INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
1026INFO: telemetry.client: Accumulated 0 events. Flush the clients.
1027INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
1028INFO: telemetry.save: Save telemetry record of length 3744 in cache
1029WARNING: telemetry.check: Negative: The /home/runner/.azure/telemetry.txt was modified at 2023-03-09 10:02:31.582632, which in less than 600.000000 s
1030Error: Process completed with exit code 1.
it says it is not authorized to access the Machine learning workspace's default blobstore while creating the job.
Please help me figuring out the issue.
Thanks in advance.
Jaiman, Amardeep (Baker Hughes Contractor) Have you used a storage account that is not in the same resource group as your workspace, container registry, key vault and app insights resources? Since Azure ML workspace creation allows existing storage resources to be used, your workspace might be in the RG for which you have given access, but the storage account might be in a different RG. If you have used the right steps to provide access through SP, I only see the scope of your assignment on RG to be causing this, since you have already used contributor role for this RG. Try changing the scope to subscription level and check if it works?
@romungi-MSFT Thanks for responding. I am using the storage account which has been created while creating the workspace only and both of them are in the same resource group.
Please find below the debug logs of the requet which is failing under the hood while creating the job:
DEBUG: urllib3.connectionpool: https://management.azure.com:443 "POST /subscriptions/***/resourceGroups/my-resource-group/providers/Microsoft.MachineLearningServices/workspaces/my-workspace/datastores/workspaceblobstore/listSecrets?api-version=2022-10-01 HTTP/1.1" 200 None
DEBUG: azure.core.pipeline.policies._universal: Response status: '200'
Response headers:
'Cache-Control': 'no-cache'
'Pragma': 'no-cache'
'Transfer-Encoding': 'chunked'
'Content-Type': 'application/json; charset=utf-8'
'Content-Encoding': 'gzip'
'Expires': '-1'
'Vary': 'Accept-Encoding'
'x-ms-ratelimit-remaining-subscription-writes': '1199'
'Request-Context': 'appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d'
'x-ms-response-type': 'standard'
'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
'X-Content-Type-Options': 'nosniff'
'x-aml-cluster': 'vienna-eastus-01'
'x-request-time': '0.387'
'Server-Timing': 'traceparent;desc="00-611142c24419c7f27f3ad85cf6c31676-206ee8cc1f2cba93-01"'
'x-ms-request-id': '9de316ce-97b2-42a6-b7c9-b2b71fde3604'
'x-ms-correlation-request-id': '9de316ce-97b2-42a6-b7c9-b2b71fde3604'
'x-ms-routing-request-id': 'EASTUS2:20230310T071341Z:9de316ce-97b2-42a6-b7c9-b2b71fde3604'
'Date': 'Fri, 10 Mar 2023 07:13:40 GMT'
Response content:
{
"secretsType": "AccountKey",
"key": "*********************************"
}
DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): stvmiamaml02dev.blob.core.windows.net:443
DEBUG: urllib3.connectionpool: https://my-datastore-name.blob.core.windows.net:443 "HEAD /azureml-blobstore-0b8f3526-1fc0-40d1-baf3-d7307b38e1d3/LocalUpload/6817a5b2ed28ec2ddb961291cedf1fe3/src/preprocess.py HTTP/1.1" 403 0
Traceback (most recent call last):
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/job.py", line 90, in ml_job_create
job = ml_client.jobs.create_or_update(job=job, skip_validation=skip_validation)
File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer
return func(*args, **kwargs)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 562, in create_or_update
raise ex
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 498, in create_or_update
self._resolve_arm_id_or_upload_dependencies(job)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 842, in _resolve_arm_id_or_upload_dependencies
self._resolve_arm_id_or_azureml_id(job, self._orchestrators.get_asset_arm_id)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1064, in _resolve_arm_id_or_azureml_id
job = self._resolve_arm_id_for_pipeline_job(job, resolver)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1179, in _resolve_arm_id_for_pipeline_job
self._component_operations._resolve_dependencies_for_pipeline_component_jobs(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 763, in _resolve_dependencies_for_pipeline_component_jobs
component_cache.resolve_nodes()
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_cache_utils.py", line 376, in resolve_nodes
self._resolve_nodes()
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_cache_utils.py", line 340, in _resolve_nodes
self._resolve_cache_contents(cache_contents_to_resolve, resolver=self._resolver)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_cache_utils.py", line 270, in _resolve_cache_contents
for cache_content, resolution_results in zip(cache_contents_to_resolve, resolution_results):
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 224, in get_asset_arm_id
result = self._get_component_arm_id(asset)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 350, in _get_component_arm_id
component._id = self._component.create_or_update(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 320, in create_or_update
self._resolve_arm_id_or_upload_dependencies(component)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 504, in _resolve_arm_id_or_upload_dependencies
_try_resolve_code_for_component(component=component, get_arm_id_and_fill_back=get_arm_id_and_fill_back)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 844, in _try_resolve_code_for_component
component.code = get_arm_id_and_fill_back(code, azureml_type=AzureMLResourceType.CODE)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 216, in get_asset_arm_id
result = self._get_code_asset_arm_id(asset, register_asset=register_asset)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 273, in _get_code_asset_arm_id
raise e
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 257, in _get_code_asset_arm_id
code_asset = self._code_assets.create_or_update(code_asset)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_code_operations.py", line 142, in create_or_update
raise ex
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_code_operations.py", line 95, in create_or_update
code, _ = _check_and_upload_path(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_artifact_utilities.py", line 401, in _check_and_upload_path
uploaded_artifact = _upload_to_datastore(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_artifact_utilities.py", line 299, in _upload_to_datastore
artifact = upload_artifact(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_artifact_utilities.py", line 178, in upload_artifact
artifact_info = storage_client.upload(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_blob_storage_helper.py", line 103, in upload
upload_directory(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_asset_utils.py", line 616, in upload_directory
storage_client.check_blob_exists()
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_blob_storage_helper.py", line 198, in check_blob_exists
raise e
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_blob_storage_helper.py", line 157, in check_blob_exists
properties = blob_client.get_blob_properties()
File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer
return func(*args, **kwargs)
File "/opt/az/azcliextensions/ml/azure/storage/blob/_blob_client.py", line 1332, in get_blob_properties
process_storage_error(error)
File "/opt/az/azcliextensions/ml/azure/storage/blob/_shared/response_handlers.py", line 189, in process_storage_error
exec("raise error from None") # pylint: disable=exec-used # nosec
File "<string>", line 1, in <module>
azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'This request is not authorized to perform this operation.'
ErrorCode:AuthorizationFailure
ERROR: cli: None
DEBUG: cli: Received HttpResponseError: Traceback (most recent call last):
As you can see in the logs, that here secrets of the storage account was been successfully fetched but while uploading the source code file in the blob store, it says
"azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'This request is not authorized to perform this operation.'"
Do you see that its an issue with Service Principal's permissions ? In that case , could you please provide the list of permissions which needs to be attached to role of service principal to successfully automate the Azure ML pipeline or is there any way to confirm if the service principals have required permissions to run the ML pipeline?
Also do you see if it is related to the Managed identity of the AML clusters have been used in the ML pipeline ?
Try adding Storage Blob Data Contributor role on your MI access since you seem to be using user assigned MI on your compute clusters where this command or job is run. Is that correct?
No I am using System Assigned Managed Identity while creating the AML cluster. Please find below the bicep template i am using to create the AML cluster.
resource trainingcluster 'Microsoft.MachineLearningServices/workspaces/computes@2022-05-01' = {
name: '${machineLearning}/${clusterName}'
location: location
identity: {
type: 'SystemAssigned'
}
properties: {
computeType: 'AmlCompute'
computeLocation: location
description: 'Machine Learning AML cluster'
So I am NOT using user assigned identity on AML cluster. What could be the reason then ?
Hi @romungi-MSFT , Could you please review the last response. We are using system assigned managed identity to clusters. The SP i am using to authorize the Pipeline has Storage Blob Data Contributor role attached to it.
We have also tried with assigning User assigned managed identity to compute cluster. and issue persists.
Any suggestions ?
Jaiman, Amardeep (Baker Hughes Contractor) Apologies for not responding sooner. I have seen the log above and it seems to only suggest that the permissions are somehow not being reflected on SP which is used by your compute cluster. I am sure you must have reviewed the principal id of your MI in Azure portal under Azure active directory to verify the permissions on the MI and the compute cluster resource. The resource id or cluster link should be available from this page to review the permissions on the cluster. Try adding the above role assignments from the cluster resource page to cross check if this helps fixing the problem.
@romungi-MSFT it seems we have resource group contributor level access on SP and UAIM with storage blob contributor role assigned on the Datastore resource. any suggestions ? shall we create any support ticket to jointly debug this issue with microsoft ?
Jaiman, Amardeep (Baker Hughes Contractor) I would recommend using the support approach as we have tried our best to check the possible causes. Support along with service team should be able to pin point any configuration or permission issues in your current setup.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 44%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERR%3C/text%3E%3C/svg%3E)
Hi @JaimanAmardeep-3234 - if you are accessing your blob via pep, then whitelist your firewall vnet/snet also on the "Enabled from selected virtual networks and IP addresses" on your storageAcc.