Using UAMI in Azure Synapse Notebooks

Question

I have a bunch of Synapse notebooks currently using AppId and secret through Linked Services and successfully running so far. As part of some initiative we need to migrate these notebooks to consuming User Assigned Managed Identity (UAMI).

Notebooks primarily uses Kusto cluster to fetch some data and process, model etc.

I'm able to create credentials and linked services using the newly created UAMIs and successfully "test connections", however while using these Linked services, i'm getting the below error and the documentation suggests that UAMI are not supported.

Can someone point me to alternatives/work arounds, to achieve the same!

Py4JJavaError: An error occurred while calling z:mssparkutils.credentials.getPropertiesAll. : com.microsoft.azure.synapse.tokenlibrary.TokenLibrary$NonRetryableStatusException$1: POST failed with 'Bad Request' (400) and message: {"result":"DependencyError","errorId":"BadRequest","errorMessage":"[Code=CredentialTypeNotSupported, Target=1_UAMI, Message=Failed to load LinkedService, Exception: Credential: 1-Kusto is of type UAMI and is not supported]. TraceId : c4174624-0872-47c9-87c2-193522ec20b5 | client-request-id : 358ea3c0-00b1-4d36-b3d9-ef4d36aa37ee. Error Component : LSR"}

Answer 1

Hello @Harishga ,

We have similar problem in our team.

We have Synapse Pipeline with Spark Job Definition
We used SPN to read data from ADLS2 and write to kusto with Spark Kusto Connector and token from SPN

We used to save SPN credentials into AKV
We have urgent requirement to move away from SPN completely.
I have been trying to use UAMI for the past few days but seems impossible
please could you suggest us a design for our purpose

Hadoop Connection:
fs.azure.account.auth.type: "OAuth"
fs.azure.account.oauth.provider.type:"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
fs.azure.account.oauth2.client.id: $app.id

fs.azure.account.oauth2.client.secret: $app.secret
fs.azure.account.oauth2.client.endpoint: $client.endpoint

Kusto Spark Connector
df.write .format("com.microsoft.kusto.spark.datasource") .option(KustoSinkOptions.KUSTO_CLUSTER, cluster) .option(KustoSinkOptions.KUSTO_DATABASE, database) .option(KustoSinkOptions.KUSTO_TABLE, tableName) .option(KustoSinkOptions.KUSTO_ACCESS_TOKEN, token)

.mode(SaveMode.Append) .save()

I tried to use fs.azure.account.oauth.provider.type: org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider and MSI Client Id but it failed (Connection socket timeout, so can't get a token)
I tried to use a custom TokenProvider using DefaultAzureCredentialBuilder but it failed as well
I tried to generate a token using DefaultAzureCredentialBuilder and pass it to Kusto Connector but it failed

Please can you help us into our design update