How to access Key Vault secrets from a Synapse Spark Notebook?

Question

I am trying to access Key Vault secrets from a Synapse Spark Notebook securely. I followed this documentation to set up a linked service with access to the Key Vault on Synapse: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/microsoft-spark-utilities?pivots=programming-language-python#configure-access-to-azure-key-vault.

I made sure the Synapse System Assigned Managed Identity has the Key Vault Secrets User role which grants it GET and LIST permissions to the secrets in the Key Vault. Moreover, the user running the notebook has Key Vault Secrets Officer access to the secrets. The Key Vault is configured to use the RBAC security model instead of access policies. After adding the permissions, I tested the linked service could connect to the Key Vault, and it connected successfully.

After all of that is set up, I tried to access the Key Vault secrets using the Credentials utilities from the Microsoft Spark Utilities, as shown here: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/microsoft-spark-utilities?pivots=programming-language-python#get-secret-using-workspace-identity

This is the exact code I am using, and the error that I get (I removed private information using *** or <input name>):

from notebookutils import mssparkutils
mssparkutils.credentials.getSecret('<azure key vault name>','<secret name>','<key vault linked service name>')

---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
Cell In [5], line 2
      1 from notebookutils import mssparkutils
----> 2 mssparkutils.credentials.getSecret('*********.vault.azure.net', '*********', '***********')

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/notebookutils/mssparkutils/credentials.py:23, in getSecret(akvName, secret, linkedService)
     21     return creds.getSecret(akvName, secret)
     22 else:
---> 23     return creds.getSecret(akvName, secret, linkedService)

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/java_gateway.py:1321, in JavaMember.__call__(self, *args)
   1315 command = proto.CALL_COMMAND_NAME +\
   1316     self.command_header +\
   1317     args_command +\
   1318     proto.END_COMMAND_PART
   1320 answer = self.gateway_client.send_command(command)
-> 1321 return_value = get_return_value(
   1322     answer, self.gateway_client, self.target_id, self.name)
   1324 for temp_arg in temp_args:
   1325     temp_arg._detach()

File /opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py:190, in capture_sql_exception.<locals>.deco(*a, **kw)
    188 def deco(*a: Any, **kw: Any) -> Any:
    189     try:
--> 190         return f(*a, **kw)
    191     except Py4JJavaError as e:
    192         converted = convert_exception(e.java_exception)

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name)
    324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325 if answer[1] == REFERENCE_TYPE:
--> 326     raise Py4JJavaError(
    327         "An error occurred while calling {0}{1}{2}.\n".
    328         format(target_id, ".", name), value)
    329 else:
    330     raise Py4JError(
    331         "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
    332         format(target_id, ".", name, value))

Py4JJavaError: An error occurred while calling z:mssparkutils.credentials.getSecret.
: com.twitter.finagle.NoBrokersAvailableException: No hosts are available for *********.vault.azure.net:443, Dtab.base=[], Dtab.local=[]. Remote Info: Not Available

I verified the Key Vault exists, that the Synapse MSI and user running the notebook can get the secrets, and the Key Vault is accessible from the Synapse workspace (I tested the linked service can connect to the Key Vault). What am I missing? Why am I getting this error? How can I get the Key Vault secrets from a Synapse Spark Notebook?

Thanks in advance!EDIT:

• Added the Key Vault uses the RBAC security model.

Answer 1

Hello @Carlos Garcia-Vaso !

The important detail is how yu have configured the Key Vault Access ?

Is it via RBAC or Access Policies

Mind this ;

Key Vault Secrets OfficerPerform any action on the secrets of a key vault, except manage permissions. Only works for key vaults that use the 'Azure role-based access control' permission model.*

Please check the way your Key Vault allows access and mak ethe correct assigmnet !

Let us know your feedback !

--

I hope this helps!

Kindly mark the answer as Accepted and Upvote in case it helped!

Regards

Answer 2

Hello @Carlos Garcia-Vaso !

Thank you

My Suggestion : Change to Access Policies and Use Entra ID Access Permissions

Have a look : https://youtu.be/03axWN3Q6zE?si=bV5y1T6FKgDLd8Mq

Please try and post your input !

--

I hope this helps!

Kindly mark the answer as Accepted and Upvote in case it helped!

Regards