Permission issue while creating a ML pipeline job using Azure ML CLI v2
Hi I am using Azure ML CLI v2 to create a ML pipeline job for training the model. using following command :
az ml job create --file ./mlops/azureml/train/train-pipeline.yml --resource-group myresource_group \
--workspace-name myworkspace --query name -o tsv
I am using service principal with contributor access on resource group level to authorize the command. But i am getting following error on running the command and ML pipeline job is failed to create
DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): stvmiamaml02dev.blob.core.windows.net:443
DEBUG: urllib3.connectionpool: https://stvmiamaml02dev.blob.core.windows.net:443 "HEAD /azureml-blobstore-0b8f3526-1fc0-40d1-baf3-d7307b38e1d3/LocalUpload/6817a5b2ed28ec2ddb961291cedf1fe3/src/preprocess.py HTTP/1.1" 403 0
Traceback (most recent call last):
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/job.py", line 90, in ml_job_create
job = ml_client.jobs.create_or_update(job=job, skip_validation=skip_validation)
File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer
return func(*args, **kwargs)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 562, in create_or_update
raise ex
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 498, in create_or_update
self._resolve_arm_id_or_upload_dependencies(job)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 842, in _resolve_arm_id_or_upload_dependencies
self._resolve_arm_id_or_azureml_id(job, self._orchestrators.get_asset_arm_id)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1064, in _resolve_arm_id_or_azureml_id
job = self._resolve_arm_id_for_pipeline_job(job, resolver)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1179, in _resolve_arm_id_for_pipeline_job
self._component_operations._resolve_dependencies_for_pipeline_component_jobs(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 763, in _resolve_dependencies_for_pipeline_component_jobs
component_cache.resolve_nodes()
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_cache_utils.py", line 376, in resolve_nodes
self._resolve_nodes()
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_cache_utils.py", line 340, in _resolve_nodes
self._resolve_cache_contents(cache_contents_to_resolve, resolver=self._resolver)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_cache_utils.py", line 270, in _resolve_cache_contents
for cache_content, resolution_results in zip(cache_contents_to_resolve, resolution_results):
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 224, in get_asset_arm_id
result = self._get_component_arm_id(asset)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 350, in _get_component_arm_id
component._id = self._component.create_or_update(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 320, in create_or_update
self._resolve_arm_id_or_upload_dependencies(component)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 504, in _resolve_arm_id_or_upload_dependencies
_try_resolve_code_for_component(component=component, get_arm_id_and_fill_back=get_arm_id_and_fill_back)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 844, in _try_resolve_code_for_component
component.code = get_arm_id_and_fill_back(code, azureml_type=AzureMLResourceType.CODE)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 216, in get_asset_arm_id
result = self._get_code_asset_arm_id(asset, register_asset=register_asset)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 273, in _get_code_asset_arm_id
raise e
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 257, in _get_code_asset_arm_id
code_asset = self._code_assets.create_or_update(code_asset)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_code_operations.py", line 142, in create_or_update
raise ex
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_code_operations.py", line 95, in create_or_update
code, _ = _check_and_upload_path(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_artifact_utilities.py", line 401, in _check_and_upload_path
uploaded_artifact = _upload_to_datastore(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_artifact_utilities.py", line 299, in _upload_to_datastore
artifact = upload_artifact(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_artifact_utilities.py", line 178, in upload_artifact
artifact_info = storage_client.upload(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_blob_storage_helper.py", line 103, in upload
upload_directory(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_asset_utils.py", line 616, in upload_directory
storage_client.check_blob_exists()
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_blob_storage_helper.py", line 198, in check_blob_exists
raise e
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_blob_storage_helper.py", line 157, in check_blob_exists
properties = blob_client.get_blob_properties()
File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer
return func(*args, **kwargs)
File "/opt/az/azcliextensions/ml/azure/storage/blob/_blob_client.py", line 1332, in get_blob_properties
process_storage_error(error)
File "/opt/az/azcliextensions/ml/azure/storage/blob/_shared/response_handlers.py", line 189, in process_storage_error
exec("raise error from None") # pylint: disable=exec-used # nosec
File "<string>", line 1, in <module>
azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'This request is not authorized to perform this operation.'
ErrorCode:AuthorizationFailure
ERROR: cli: None
DEBUG: cli: Received HttpResponseError: Traceback (most recent call last):
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/job.py", line 90, in ml_job_create
job = ml_client.jobs.create_or_update(job=job, skip_validation=skip_validation)
File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer
return func(*args, **kwargs)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 562, in create_or_update
raise ex
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 498, in create_or_update
self._resolve_arm_id_or_upload_dependencies(job)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 842, in _resolve_arm_id_or_upload_dependencies
self._resolve_arm_id_or_azureml_id(job, self._orchestrators.get_asset_arm_id)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1064, in _resolve_arm_id_or_azureml_id
job = self._resolve_arm_id_for_pipeline_job(job, resolver)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1179, in _resolve_arm_id_for_pipeline_job
self._component_operations._resolve_dependencies_for_pipeline_component_jobs(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 763, in _resolve_dependencies_for_pipeline_component_jobs
component_cache.resolve_nodes()
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_cache_utils.py", line 376, in resolve_nodes
self._resolve_nodes()
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_cache_utils.py", line 340, in _resolve_nodes
self._resolve_cache_contents(cache_contents_to_resolve, resolver=self._resolver)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_cache_utils.py", line 270, in _resolve_cache_contents
for cache_content, resolution_results in zip(cache_contents_to_resolve, resolution_results):
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 224, in get_asset_arm_id
result = self._get_component_arm_id(asset)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 350, in _get_component_arm_id
component._id = self._component.create_or_update(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 320, in create_or_update
self._resolve_arm_id_or_upload_dependencies(component)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 504, in _resolve_arm_id_or_upload_dependencies
_try_resolve_code_for_component(component=component, get_arm_id_and_fill_back=get_arm_id_and_fill_back)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 844, in _try_resolve_code_for_component
component.code = get_arm_id_and_fill_back(code, azureml_type=AzureMLResourceType.CODE)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 216, in get_asset_arm_id
result = self._get_code_asset_arm_id(asset, register_asset=register_asset)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 273, in _get_code_asset_arm_id
raise e
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 257, in _get_code_asset_arm_id
code_asset = self._code_assets.create_or_update(code_asset)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_code_operations.py", line 142, in create_or_update
raise ex
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_code_operations.py", line 95, in create_or_update
code, _ = _check_and_upload_path(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_artifact_utilities.py", line 401, in _check_and_upload_path
uploaded_artifact = _upload_to_datastore(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_artifact_utilities.py", line 299, in _upload_to_datastore
artifact = upload_artifact(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_artifact_utilities.py", line 178, in upload_artifact
artifact_info = storage_client.upload(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_blob_storage_helper.py", line 103, in upload
upload_directory(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_asset_utils.py", line 616, in upload_directory
storage_client.check_blob_exists()
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_blob_storage_helper.py", line 198, in check_blob_exists
raise e
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_blob_storage_helper.py", line 157, in check_blob_exists
properties = blob_client.get_blob_properties()
File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer
return func(*args, **kwargs)
File "/opt/az/azcliextensions/ml/azure/storage/blob/_blob_client.py", line 1332, in get_blob_properties
process_storage_error(error)
File "/opt/az/azcliextensions/ml/azure/storage/blob/_shared/response_handlers.py", line 189, in process_storage_error
exec("raise error from None") # pylint: disable=exec-used # nosec
File "<string>", line 1, in <module>
azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'This request is not authorized to perform this operation.'
ErrorCode:AuthorizationFailure
DEBUG: cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/opt/az/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
raise ex
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
result = cmd_copy(params)
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
return self.handler(*args, **kwargs)
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
return op(**command_args)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/job.py", line 104, in ml_job_create
log_and_raise_error(err, debug, yaml_operation=yaml_operation)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/raise_error.py", line 181, in log_and_raise_error
raise cli_error
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/job.py", line 90, in ml_job_create
job = ml_client.jobs.create_or_update(job=job, skip_validation=skip_validation)
File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer
return func(*args, **kwargs)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 562, in create_or_update
raise ex
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 498, in create_or_update
self._resolve_arm_id_or_upload_dependencies(job)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 842, in _resolve_arm_id_or_upload_dependencies
self._resolve_arm_id_or_azureml_id(job, self._orchestrators.get_asset_arm_id)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1064, in _resolve_arm_id_or_azureml_id
job = self._resolve_arm_id_for_pipeline_job(job, resolver)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1179, in _resolve_arm_id_for_pipeline_job
self._component_operations._resolve_dependencies_for_pipeline_component_jobs(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 763, in _resolve_dependencies_for_pipeline_component_jobs
component_cache.resolve_nodes()
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_cache_utils.py", line 376, in resolve_nodes
self._resolve_nodes()
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_cache_utils.py", line 340, in _resolve_nodes
self._resolve_cache_contents(cache_contents_to_resolve, resolver=self._resolver)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_cache_utils.py", line 270, in _resolve_cache_contents
for cache_content, resolution_results in zip(cache_contents_to_resolve, resolution_results):
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 224, in get_asset_arm_id
result = self._get_component_arm_id(asset)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 350, in _get_component_arm_id
component._id = self._component.create_or_update(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 320, in create_or_update
self._resolve_arm_id_or_upload_dependencies(component)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 504, in _resolve_arm_id_or_upload_dependencies
_try_resolve_code_for_component(component=component, get_arm_id_and_fill_back=get_arm_id_and_fill_back)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_component_operations.py", line 844, in _try_resolve_code_for_component
component.code = get_arm_id_and_fill_back(code, azureml_type=AzureMLResourceType.CODE)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 216, in get_asset_arm_id
result = self._get_code_asset_arm_id(asset, register_asset=register_asset)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 273, in _get_code_asset_arm_id
raise e
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 257, in _get_code_asset_arm_id
code_asset = self._code_assets.create_or_update(code_asset)
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_code_operations.py", line 142, in create_or_update
raise ex
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_code_operations.py", line 95, in create_or_update
code, _ = _check_and_upload_path(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_artifact_utilities.py", line 401, in _check_and_upload_path
uploaded_artifact = _upload_to_datastore(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_artifact_utilities.py", line 299, in _upload_to_datastore
artifact = upload_artifact(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_artifact_utilities.py", line 178, in upload_artifact
artifact_info = storage_client.upload(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_blob_storage_helper.py", line 103, in upload
upload_directory(
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_asset_utils.py", line 616, in upload_directory
storage_client.check_blob_exists()
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_blob_storage_helper.py", line 198, in check_blob_exists
raise e
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_artifacts/_blob_storage_helper.py", line 157, in check_blob_exists
properties = blob_client.get_blob_properties()
File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer
return func(*args, **kwargs)
File "/opt/az/azcliextensions/ml/azure/storage/blob/_blob_client.py", line 1332, in get_blob_properties
process_storage_error(error)
File "/opt/az/azcliextensions/ml/azure/storage/blob/_shared/response_handlers.py", line 189, in process_storage_error
exec("raise error from None") # pylint: disable=exec-used # nosec
File "<string>", line 1, in <module>
azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'This request is not authorized to perform this operation.'
ErrorCode:AuthorizationFailure
ERROR: cli.azure.cli.core.azclierror: Operation returned an invalid status 'This request is not authorized to perform this operation.'
ErrorCode:AuthorizationFailure
ERROR: az_command_data_logger: Operation returned an invalid status 'This request is not authorized to perform this operation.'
ErrorCode:AuthorizationFailure
DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f6f3d7708b0>]
INFO: az_command_data_logger: exit code: 1
INFO: cli.__main__: Command ran in 8.520 seconds (init: 0.336, invoke: 8.184)
INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
INFO: telemetry.client: Accumulated 0 events. Flush the clients.
INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
INFO: telemetry.save: Save telemetry record of length 3744 in cache
WARNING: telemetry.check: Negative: The /home/runner/.azure/telemetry.txt was modified at 2023-03-09 10:02:31.582632, which in less than 600.000000 s
Error: Process completed with exit code 1.
As per the above logs, its trying to access the machine learning workspace default blobstore but getting authorization error. The AML clusters used in the ML pipeline have been create using same Service principals and assigned Managed identity.
Please help me figuring out the issue. please let me know for any additional information.