ADF is not able to access Keyvault secrests via ADB notebook

Amardeep Aklujkar 26

We have stored credentials in Azure keyvault we have provided access to both ADF and ADB to access keyvault details.(Get,List). we are able to access secret directly from ADF if we create sample pipeline to check the access. we are able to access secret directly in ADB if we run notebook manually.

Issue: we are trying to run ADB notebook [ in notebook we are mentioning secretscope details and secret details] from ADF. But we are getting an error of Unauthorized access.

com.databricks.common.client.UnexpectedHttpError: HTTP request failed with status: HTTP/1.1 401 Unauthorized

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2023-01-10T23:42:08.157+00:00
Hello @Amardeep Aklujkar ,

Thanks for the question and using MS Q&A platform.

Could you please confirm if you have enabled Enable admin protection for “No isolation shared” clusters on your account feature across the workspace. If yes, then that could be the reason why jobs are failing as it was hitting limitation with this admin protection. For more info about the limitation please refer to this document: Enable the account-level admin protection setting - Limitations

Then, to overcome this issue, Azure Databricks recommends that admins do one of the following:

Use a different cluster type other than “No isolation shared” cluster access type or its equivalent legacy cluster types.

Create a non-admin user when using No Isolation Shared clusters.

Hope this info helps. Do let us know how it goes.

Thank you
Amardeep Aklujkar 26 Reputation points

2023-01-11T11:50:32.8866667+00:00

Thanks for the update!

Above option is not working.

We are trying to run the job using new job cluster every time.
Amardeep Aklujkar 26 Reputation points

2023-01-12T07:36:08.17+00:00

@KranthiPakala-MSFT Can you please check this once.
Amardeep Aklujkar 26 Reputation points

2023-01-12T23:03:48.21+00:00

Thanks for the update!

Above option is not working.

We are trying to run the job using new job cluster every time.
karthik elavan 0 Reputation points

2023-02-03T23:01:01.14+00:00

Dear All,

We are faced similar issues and fixed in Dev environment with help of to upgraded our one time job cluster version which is 11.2X but if we moved similar kind if version and permission, we are getting similar issues throwing,

"runError": "com.databricks.common.client.UnexpectedHttpError: HTTP request failed with status: HTTP/1.1 401 Unauthorized",

Kindly advice.
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2023-02-03T23:04:13.7766667+00:00

Hi Amardeep Aklujkar, karthik elavan, I tried to reproduce the issue on my end but no luck, everything was running fine.

I'm suspecting something missing on user configuration part, we need to evaluate series of steps involved to identify the root cause. Hence, I would recommend filing a support ticket for deeper investigation and please do share the SR# once it is filed so that we can't rack it internally. In case if you don't have a support plan, please let me know here, so that I can work offline for enabling a one-time free support ticket for you.

Thanks