An error occurred when using odbc and managed identity to link azure databricks in the. net core

ZhiHong Zhang (BEYONDSOFT CONSULTING INC) 0

Environmental description:

The app service has already been configured with managed identity, and the relevant permissions for this managed identity have been configured in Azure databricks. When the app service uses code to access Azure databricks, the following error message will occur：

ERROR [28000] [Simba][DriverOAuthSupport] (8719) Can't access to Azure Metadata Service, check if you run on VM with MI: OAuthAPIErr

User's image

According to the reference document, the use of the app service failed

The reference document states that Azure VM needs to be configured, but how to use managed identity to authorize access and use Azure databricks in the app service.

https://learn.microsoft.com/en-us/azure/databricks/integrations/odbc/authentication

PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee

2024-04-23T09:53:26.91+00:00

@ZhiHong Zhang (BEYONDSOFT CONSULTING INC) - Thanks for the question and using MS Q&A platform.

It seems like you are facing an issue while using ODBC and managed identity to link Azure Databricks in .NET Core. The error message you are receiving indicates that there is an issue with accessing the Azure Metadata Service.

To use managed identity to authorize access and use Azure Databricks in the app service, you need to follow the steps mentioned in the reference document you shared.

However, I would like to clarify that the reference document you shared is for configuring ODBC authentication for Azure Databricks. If you are not using ODBC, you may not need to follow all the steps mentioned in the document.

To use managed identity to access Azure Databricks, you can follow the steps mentioned in the Azure Databricks documentation. You can use either system-assigned or user-assigned managed identity authentication.

For system-assigned managed identity authentication, you need to retrieve the managed identity information and grant the managed identity the correct permissions in Azure Databricks. In general, you must grant at least the Contributor role to your system-assigned managed identity in Access control (IAM) of Azure Databricks.

For user-assigned managed identity authentication, you need to create one or multiple user-assigned managed identities and grant permission in your Azure Databricks. In general, you must grant at least the Contributor role to your user-assigned managed identity in Access control (IAM) of Azure Databricks.

After you have granted the correct permissions, you can create credentials for each user-assigned managed identity and use them to authenticate your app service to Azure Databricks.

I hope this helps! Let me know if you have any further questions.
ZhiHong Zhang (BEYONDSOFT CONSULTING INC) 0 Reputation points

2024-04-24T07:23:48.44+00:00

As you mentioned above, the basic permission settings have been determined to be set according to the document. After some testing, the same issue was still found.

But starting from step 5 in the document, it is necessary to configure Azure VM, but the current condition is that it is in the web app service, not Azure VM. And the current operating environment is. net core. All ODBC versions in the document must be at least 2.7.7. This is completely certain. When excluding external conditions and running in the code, I even take out Auth-ClientId and repeatedly use the set value. Then you will get this error:

Error [28000] [Simba] [DriverOAuthSupport] (8719) Can't access Azure metadata service, check if you run on VM with MI: OAuthAPIErr

And the speed is very fast, and the error is clearly pointed out as running in the VM. However, we are currently using a web app service and deploying it in a container style. Based on my guess, this authentication method did not recognize MI at all, so it immediately returned an exception.

I don't know if there are any examples that can be provided to me to correctly access Azure databricks and use query statements in the. net core environment of C # code. If there are no examples, would you like to inform me that they are not currently supported. Because there is a reference below indicating that some SDKs do not support it.

https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/odbc/2.8.0/SimbaSparkODBC-2.8.0.1002-Debian-64bit.zip

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/azure-mi-auth

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/
ZhiHong Zhang (BEYONDSOFT CONSULTING INC) 0 Reputation points

2024-04-24T07:26:08.2966667+00:00

As you mentioned above, the basic permission settings have been determined to be set according to the document. After some testing, the same issue was still found.

But starting from step 5 in the document, it is necessary to configure Azure VM, but the current condition is that it is in the web app service, not Azure VM. And the current operating environment is. net core. All ODBC versions in the document must be at least 2.7.7. This is completely certain. When excluding external conditions and running in the code, I even take out Auth-ClientId and repeatedly use the set value. Then you will get this error:

Error [28000] [Simba] [DriverOAuthSupport] (8719) Can't access Azure metadata service, check if you run on VM with MI: OAuthAPIErr

And the speed is very fast, and the error is clearly pointed out as running in the VM. However, we are currently using a web app service and deploying it in a container style. Based on my guess, this authentication method did not recognize MI at all, so it immediately returned an exception.

I don't know if there are any examples that can be provided to me to correctly access Azure databricks and use query statements in the. net core environment of C # code. If there are no examples, would you like to inform me that they are not currently supported. Because there is a reference below indicating that some SDKs do not support it.

https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/odbc/2.8.0/SimbaSparkODBC-2.8.0.1002-Debian-64bit.zip

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/azure-mi-auth

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/
PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee

2024-04-25T05:18:33.11+00:00

@ZhiHong Zhang (BEYONDSOFT CONSULTING INC) - I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?
PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee

2024-04-29T03:04:24.8633333+00:00

@ZhiHong Zhang (BEYONDSOFT CONSULTING INC) - Did you get a chance to open a support ticket?
ZhiHong Zhang (BEYONDSOFT CONSULTING INC) 0 Reputation points

2024-04-30T02:56:45.0233333+00:00

@PRADEEPCHEEKATLA-MSFT The ticket id is 2404250060001658. I have already submitted it, and the first response I gave was still the link in the document I provided. I followed the instructions in the document, the only difference being that I used a web app instead of a VM. However, the document states that 'such as VM' is theoretically working, but in reality it is not.
PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee

2024-04-30T03:22:16.7133333+00:00

@ZhiHong Zhang (BEYONDSOFT CONSULTING INC) - Thanks for creating a support ticket. Once you resolve the issue with the help of support, kindly share the solution, which might be beneficial to other community members reading this thread.

1 answer

ZhiHong Zhang (BEYONDSOFT CONSULTING INC) 0 Reputation points

2024-04-24T07:23:23.6366667+00:00

As you mentioned above, the basic permission settings have been determined to be set according to the document. After some testing, the same issue was still found.

But starting from step 5 in the document, it is necessary to configure Azure VM, but the current condition is that it is in the web app service, not Azure VM. And the current operating environment is. net core. All ODBC versions in the document must be at least 2.7.7. This is completely certain. When excluding external conditions and running in the code, I even take out Auth-ClientId and repeatedly use the set value. Then you will get this error:

Error [28000] [Simba] [DriverOAuthSupport] (8719) Can't access Azure metadata service, check if you run on VM with MI: OAuthAPIErr

And the speed is very fast, and the error is clearly pointed out as running in the VM. However, we are currently using a web app service and deploying it in a container style. Based on my guess, this authentication method did not recognize MI at all, so it immediately returned an exception.

I don't know if there are any examples that can be provided to me to correctly access Azure databricks and use query statements in the. net core environment of C # code. If there are no examples, would you like to inform me that they are not currently supported. Because there is a reference below indicating that some SDKs do not support it.

https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/odbc/2.8.0/SimbaSparkODBC-2.8.0.1002-Debian-64bit.zip

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/azure-mi-auth

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/
Please sign in to rate this answer.

0 comments No comments
Sign in to comment