Using SMI in Synapse Spark Job

Jean-Christian Kouamé 0 Microsoft Employee

Hello

We have Synapse Pipeline with Spark Job Definition We currently use SPN to read data from ADLS2 and also to write to kusto with Spark Kusto Connector (Getting a token from SPN)

We have saved SPN credentials into AKV.

We have urgent requirement to move away from SPN completely. I have been trying to use UAMI for the past few days but seems impossible please could you suggest us a design for our purpose

Hadoop Connection:
fs.azure.account.auth.type: "OAuth"
fs.azure.account.oauth.provider.type:"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
fs.azure.account.oauth2.client.id: $app.id

fs.azure.account.oauth2.client.secret: $app.secret fs.azure.account.oauth2.client.endpoint: $client.endpoint

Kusto Spark Connector:
df.write .format("com.microsoft.kusto.spark.datasource") .option(KustoSinkOptions.KUSTO_CLUSTER, cluster) .option(KustoSinkOptions.KUSTO_DATABASE, database) .option(KustoSinkOptions.KUSTO_TABLE, tableName) .option(KustoSinkOptions.KUSTO_ACCESS_TOKEN, token)

.mode(SaveMode.Append) .save()

For hadoop, I tried to use fs.azure.account.oauth.provider.type: org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider and MSI Client Id but it failed (Connection socket timeout, so can't get a token) I tried to use a custom TokenProvider using DefaultAzureCredentialBuilder but it failed.
Similarly, I tried to generate a token using DefaultAzureCredentialBuilder and pass it to Kusto Connector but it failed

Please can you help us into our design update, with no SPN involved

phemanth 8,645 Reputation points Microsoft Vendor

2024-05-30T06:06:16.33+00:00
@Jean-Christian Kouamé

Thanks for using MS Q&A platform and posting your query

Here's a design update to remove SPN from your Synapse Pipeline with Spark Jobs for accessing ADLS2 and Kusto:

1. Managed Identity for Azure Resources (Msi):

Enable Managed Identity for your Synapse Workspace in Azure Active Directory (AAD). This grants the workspace an identity within AAD that can be used for authentication.

Update your Hadoop connection properties: fs.azure.account.auth.type: "MSI" Remove fs.azure.account.oauth.provider.type and related properties.

2. Azure Databricks TokenProvider (Optional):

If Msi doesn't work for some reason, consider using Azure Databricks TokenProvider. This requires setting up a Databricks workspace, but it provides a more robust token acquisition mechanism.

Follow the configuration steps in the official documentation: https://docs.databricks.com/en/dev-tools/auth/index.html

3. Kusto Spark Connector with MSI:

The Kusto Spark Connector v2.3.1+ supports Managed Identity for authentication.

Update your code to leverage the KustoConnectionStringBuilder class:

from com.microsoft.kusto.spark.datasource.impl.kustoconnectionstringbuilder import KustoConnectionStringBuilder # Get the connection string from Azure Key Vault (AKV) without SPN connectionString = dbutils.secrets.get(scope="your-akv-scope", key="your-kusto-connection-string") # Build connection string with MSI connectionStringBuilder = KustoConnectionStringBuilder.withConnectionString(connectionString) connectionStringBuilder.setAadClientId("<your-workspace-msi-clientId>") # Get from Azure portal # Use the connection string for writing to Kusto df.write \ .format("com.microsoft.kusto.spark.datasource") \ .option(KustoSinkOptions.KUSTO_CONNECTION_STRING, connectionStringBuilder.toString()) \ .option(KustoSinkOptions.KUSTO_TABLE, tableName) \ .mode(SaveMode.Append) \ .save()

Troubleshooting:

Connection Socket Timeout (Msi): Ensure your Synapse Workspace has access to access data in ADLS2 and write to the Kusto cluster. Verify network connectivity and firewall rules.

Custom TokenProvider Issues: Double-check the configuration of DefaultAzureCredentialBuilder. Make sure it points to the correct MSI endpoint

Hope this helps. Do let us know if you any further queries.
Jean-Christian Kouamé 0 Reputation points Microsoft Employee

2024-05-30T16:16:03.77+00:00
Hello @phemanth ,

Thank you for taking the time to answer my questions.

Do the approach you proposed work with both UAMI (User Assigned Managed Identity) and SAMI (System Assigned Managed Identity)

Can i deduce that MSI need to grant access to AKV, ADLS2, and Maybe Kusto as well?

if UAMI is possible, does it necessarily need to be bind to the workspace?

if UAMI is not possible:

Let's say SAMI is compromise, do this means we would need to recreate everything or we can create a new SAMI for the same workspace?

If we have multiple jobs, do this means we need a workspace for each of them to ensure one id don't have access to all DBs?

| "Update your Hadoop connection properties: fs.azure.account.auth.type: "MSI" Remove fs.azure.account.oauth.provider.type and related properties."
I don't see this option according to following documentation or do you mean use "org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider" https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/site/markdown/abfs.md

| "Verify network connectivity and firewall rules."
Based on the documentation shared above, i believe the server issuing the token is the following. Are you saying we should be able to get token with the right settings. Any idea where a firewall could have been set?
phemanth 8,645 Reputation points Microsoft Vendor

2024-05-31T03:53:18.5466667+00:00
@Jean-Christian Kouamé

Compatibility with UAMI and SAMI

The approach works with both User Assigned Managed Identity (UAMI) and System Assigned Managed Identity (SAMI) for Synapse Workspace access to ADLS2 and Kusto.

MSI Permissions

Yes, you deduced correctly. MSI (UAMI or SAMI) assigned to your Synapse Workspace needs access to:

Azure Key Vault (AKV): Permissions to retrieve secrets (connection strings) for Kusto and potentially other resources.

ADLS Gen2 storage account: Read/Write permissions depending on your Spark Job's needs for accessing data in ADLS2.

Kusto cluster: Write access (typically) for the Spark Job to write data to the Kusto cluster.

UAMI Binding

UAMI doesn't necessarily need to be bound directly to the Synapse Workspace. You can assign the UAMI to a resource group containing the workspace, or grant access at the subscription level. However, binding it to the workspace simplifies permission management.

SAMI Compromise

If a SAMI assigned to your Synapse Workspace is compromised, you don't necessarily need to recreate everything. You can:

Rotate the access keys associated with the compromised SAMI. This invalidates any existing tokens and forces re-authentication.

Assign a new SAMI to the Synapse Workspace and grant it the necessary permissions.

Multiple Workspaces

You typically don't need a separate workspace for each Spark Job. A single workspace with a single SAMI can be used for multiple jobs as long as the jobs don't require strict isolation between their access permissions. However, if you need fine-grained control over access for specific jobs, consider separate workspaces with dedicated SAMIs.

Hadoop Configuration Clarification

You're right, the documentation you linked doesn't explicitly mention "fs.azure.account.auth.type: "MSI". This property might be specific to a particular Spark library or configuration. However, the concept remains the same. You should leverage the Spark configuration options that support MSI authentication for your specific Spark environment.

Here are some resources that might be helpful:

Spark on Azure Synapse Analytics: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/microsoft-spark-utilities

Kusto Spark Connector: https://github.com/Azure/azure-kusto-spark (See the configuration section for using MSI)

Firewall Rules

The server issuing the token depends on your chosen approach:

MSI: The token is typically issued by Azure Active Directory (AAD). Firewalls on the Synapse Workspace's virtual network or within AAD itself could potentially block communication and prevent token acquisition.

Azure Databricks TokenProvider (if used): The token might be issued by the Azure Databricks workspace. Firewalls on the Databricks workspace's virtual network could be relevant.
phemanth 8,645 Reputation points Microsoft Vendor

2024-06-03T05:38:05.2333333+00:00

@Jean-Christian Kouamé We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Jean-Christian Kouamé 0 Reputation points Microsoft Employee

2024-06-03T16:58:06.2833333+00:00

Hello @phemanth ,

please see the following screenshot from https://learn.microsoft.com/en-us/azure/synapse-analytics/synapse-service-identity. It invalidates a bit what you suggested. Based on my tests, generating a MSAL token for SMI (UAMI or SAMI) fails in Synapse job, only working thing so far is using Linked Service. Please, if this issue is resolved, can you help us in bypass the issue
phemanth 8,645 Reputation points Microsoft Vendor

2024-06-04T06:48:12.3233333+00:00
@Jean-Christian Kouamé

The image you sent confirms that currently generating an MSAL token for Synapse notebooks and Spark job definitions is not supported. It specifically mentions Synapse notebooks and Spark job definitions, and suggests using Linked Services or the mssparkutils APIs instead.

Here's a breakdown of the information in the image:

Note: Synapse notebooks and Spark job definitions only support the use of system-assigned managed identity (MSI) through linked services and the mssparkutils APIs. MSAL and other authentication libraries cannot use the system-assigned managed identity.

This aligns with what you mentioned in your previous query, where you stated that your tests showed generating a MSAL token for SMI (UAMI or SAMI) fails in Synapse jobs.

While the documentation doesn't mention a specific resolution timeline, it does provide alternative approaches that you can leverage in the meantime:

Linked Services: Use Linked Services to connect to your data sources (ADLS2 and Kusto) in your Synapse Pipeline. Linked Services can be configured with MSI authentication, eliminating the need for SPNs.

mssparkutils APIs (if applicable): If your Spark code interacts with data sources directly, explore using the mssparkutils APIs for Spark jobs. These APIs might provide functionalities to leverage MSI for authentication.

Here are some additional resources that you might find helpful:

Create Linked Service with MSI: https://learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-msi

Azure Synapse Analytics Spark Utilities: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-a

Share via

Using SMI in Synapse Spark Job