Using SMI in Synapse Spark Job

Jean-Christian Kouamé 0 Reputation points Microsoft Employee
2024-05-29T14:30:25.29+00:00

Hello

We have Synapse Pipeline with Spark Job Definition We currently use SPN to read data from ADLS2 and also to write to kusto with Spark Kusto Connector (Getting a token from SPN)

We have saved SPN credentials into AKV.

We have urgent requirement to move away from SPN completely. I have been trying to use UAMI for the past few days but seems impossible please could you suggest us a design for our purpose

Hadoop Connection:
fs.azure.account.auth.type: "OAuth"
fs.azure.account.oauth.provider.type:"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
fs.azure.account.oauth2.client.id: $app.id

fs.azure.account.oauth2.client.secret: $app.secret fs.azure.account.oauth2.client.endpoint: $client.endpoint

Kusto Spark Connector:
df.write .format("com.microsoft.kusto.spark.datasource") .option(KustoSinkOptions.KUSTO_CLUSTER, cluster) .option(KustoSinkOptions.KUSTO_DATABASE, database) .option(KustoSinkOptions.KUSTO_TABLE, tableName) .option(KustoSinkOptions.KUSTO_ACCESS_TOKEN, token)

.mode(SaveMode.Append) .save()

For hadoop, I tried to use fs.azure.account.oauth.provider.type: org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider and MSI Client Id but it failed (Connection socket timeout, so can't get a token) I tried to use a custom TokenProvider using DefaultAzureCredentialBuilder but it failed.
Similarly, I tried to generate a token using DefaultAzureCredentialBuilder and pass it to Kusto Connector but it failed

Please can you help us into our design update, with no SPN involved

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,565 questions
{count} votes