Can't access to a Dataset on Azure Machine Learning with SparkPool

Question

Can't access to a Dataset on Azure Machine Learning with SparkPool

Recently we added a Synapse SparkPool to a Machine Learning to use it as "Compute / Attached computers".

When we use a Dataset to connect to Azure Storage Datalake Gen2 we have the following error:

"Resolving access token for scope "https://storage.azure.com/.default" using identity of type "MANAGED". No identity was found on compute."

Ex of the Dataset use:

ds = Dataset.get_by_name(ws, "XXXXXXXX")

If we don't use a Dataset and instead we connect directly to Azure Storage Data Lake Gen2 in the Notebook using the same compute instance (Spark Pool) the job executes succefully.

Ex of the direct connection:

%%pyspark

STRUCTURED_PATH = 'abfss://******@xxxx.dfs.core.windows.net'

df_xxxxxx = spark.read.load(STRUCTURED_PATH+'/xxxx/xxxx.snappy.parquet', format='parquet')

Answer 1

Hello Guillermo Paez

Sorry we have not heard from you. I hope you have solved this issue. For this issue, I would highly recommend you to check which SDK version you are using. Since Data Assets is a new feature in Azure Machine Learning SDK V2 and Dataset is for V1, there are a few changes you may want to take a look - https://learn.microsoft.com/en-us/azure/machine-learning/migrate-to-v2-assets-data It also contains the recommended way to call the tabular dataset/data asset in both V1 and V2. Could you please take a look?

And the whole guidance for Data Assets in V2 in case you need it. - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-create-data-assets?tabs=cli

Please check above and let me know if you are still blocked by this issue, happy to help further. I hope this helps.

Regards,

Yutong

-Please kindly accept the answer and vote 'Yes' if you feel helpful to support the community, thanks a lot.