Azure Synapse mssparkutils.credentials.getSecretWithLS started failing on parsing Key Vault URL.

Question

Azure Synapse mssparkutils.credentials.getSecretWithLS started failing on parsing Key Vault URL.

Nate Cotton 20

It appears that the implementation on how the linked service is handled has been modified an is no longer properly parsing the Key Vault URL. By messing around with other methods coming off of mssparkutils.credentials I'm able to retrieve the desired secret, but that requires providing specificity that should be coming from the linked service. I've also gone to the linked service and tested the connection and it was successful (just trying to rule out other concerns).

Executing code:

def get_secret(secret_name):
    return mssparkutils.credentials.getSecretWithLS('AzureKeyVault', secret_name)

get_secret('...') # <-- simplified

Stacktrace:

Cell In [21], line 5, in get_secret(secret_name)
      3 @cache
      4 def get_secret(secret_name):
----> 5     return mssparkutils.credentials.getSecretWithLS('AzureKeyVault', secret_name)

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/notebookutils/mssparkutils/credentials.py:26, in getSecretWithLS(linkedService, secret)
     25 def getSecretWithLS(linkedService, secret):
---> 26     return creds.getSecretWithLS(linkedService, secret)

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/java_gateway.py:1321, in JavaMember.__call__(self, *args)
   1315 command = proto.CALL_COMMAND_NAME +\
   1316     self.command_header +\
   1317     args_command +\
   1318     proto.END_COMMAND_PART
   1320 answer = self.gateway_client.send_command(command)
-> 1321 return_value = get_return_value(
   1322     answer, self.gateway_client, self.target_id, self.name)
   1324 for temp_arg in temp_args:
   1325     temp_arg._detach()

File /opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py:190, in capture_sql_exception.<locals>.deco(*a, **kw)
    188 def deco(*a: Any, **kw: Any) -> Any:
    189     try:
--> 190         return f(*a, **kw)
    191     except Py4JJavaError as e:
    192         converted = convert_exception(e.java_exception)

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name)
    324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325 if answer[1] == REFERENCE_TYPE:
--> 326     raise Py4JJavaError(
    327         "An error occurred while calling {0}{1}{2}.\n".
    328         format(target_id, ".", name), value)
    329 else:
    330     raise Py4JError(
    331         "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
    332         format(target_id, ".", name, value))

Py4JJavaError: An error occurred while calling z:mssparkutils.credentials.getSecretWithLS.
: java.lang.ArrayIndexOutOfBoundsException: 1
	at com.microsoft.azure.synapse.tokenlibrary.TokenLibrary.parseAkvUrl(TokenLibrary.scala:665)
	at com.microsoft.azure.synapse.tokenlibrary.TokenLibrary.$anonfun$getAkvSecretWithLS$1(TokenLibrary.scala:791)
	at com.twitter.util.Future.$anonfun$flatMap$1(Future.scala:1808)
	at com.twitter.util.Promise$FutureTransformer.liftedTree1$1(Promise.scala:240)
	at com.twitter.util.Promise$FutureTransformer.k(Promise.scala:240)
	at com.twitter.util.Promise$Transformer.apply(Promise.scala:215)
	at com.twitter.util.Promise$WaitQueue.com$twitter$util$Promise$WaitQueue$$run(Promise.scala:91)
	at com.twitter.util.Promise$WaitQueue$$anon$1.run(Promise.scala:86)
	at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:198)
	at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:157)
	at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:274)
	at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:109)
	at com.twitter.util.Promise$WaitQueue.runInScheduler(Promise.scala:86)
	at com.twitter.util.Promise.continue(Promise.scala:813)
	at com.twitter.util.Promise$Responder.transform(Promise.scala:332)
	at com.twitter.util.Promise$Responder.transform$(Promise.scala:329)
	at com.twitter.util.Promise.transform(Promise.scala:449)
	at com.twitter.util.Future.flatMap(Future.scala:1807)
	at com.microsoft.azure.synapse.tokenlibrary.TokenLibrary.getAkvSecretWithLS(TokenLibrary.scala:788)
	at com.microsoft.azure.synapse.tokenlibrary.TokenLibrary$.getSecretWithLS(TokenLibrary.scala:1459)
	at mssparkutils.credentials$.getSecretWithLS(credentials.scala:108)
	at mssparkutils.credentials.getSecretWithLS(credentials.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:750)

Accepted answer

0 additional answers

Your answer

Answer 1

PRADEEPCHEEKATLA 90,646 Moderator

@Nate Cotton - Thanks for the question and using MS Q&A platform.

I'm sorry to hear that you are facing issues with mssparkutils.credentials.getSecretWithLS method. It seems that the implementation on how the linked service is handled has been modified and is no longer properly parsing the Key Vault URL. However, you can try using other methods coming off of mssparkutils.credentials to retrieve the desired secret.

For example, you can use mssparkutils.credentials.getSecret method to retrieve the secret by providing the Azure Key Vault name and secret name as parameters. Here is an example code snippet in Python:

def get_secret(
    return mssparkutils.credentials.getSecretWithLS('AzureKeyVaultName', '

get_secret('...') # <-- simplified

Please replace azure-key-vault-name with the name of your Azure Key Vault and secret-name with the name of the secret you want to retrieve.

User's image

For more details, refer to Secure credentials with linked services using the mssparkutils.

In case, if you are still experiencing the same issue. I would request you to double check below requirements:

Azure Keyvalut linked service connection.
Make sure you had published all the changes to the workspace.
Make sure to pass the correct key vault name and the secret name without any spelling mistakes.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Nate Cotton 20 Reputation points

2024-03-26T22:31:08.6033333+00:00

Is there a timeline for the correction of the URL parsing? I'd prefer not to introduce additional infrastructural/configuration changes to support the other options. Yes I have confirmed that the other getSecret options do work as expected.
PRADEEPCHEEKATLA 90,646 Reputation points Moderator

2024-03-27T07:49:01.1033333+00:00
@Nate Cotton - Currently, we don't have any information about the timeline for the correction of the URL parsing issue. However, you can try using the mssparkutils.credentials.getSecret method to retrieve the desired secret. This method retrieves the secret using user credentials and does not require any additional infrastructure or configuration changes.

Here is an example of how to use the mssparkutils.credentials.getSecret method as shown above:

def get_secret(secret_name): return mssparkutils.credentials.getSecret('azure key vault name', secret_name) get_secret('secret name')

Please replace azure key vault name and secret name with the actual names of your Azure Key Vault and secret respectively.

Hope this helps. Do let us know if you any further queries.
Nate Cotton 20 Reputation points

2024-03-27T23:49:56.4133333+00:00

Given the multiple deployment environments there will be additional configuration needed. So it isn't as simple of fix, but we'll go through the effort of the correction given the unknown timeline.
PRADEEPCHEEKATLA 90,646 Reputation points Moderator

2024-03-28T05:06:49.4433333+00:00

@Nate Cotton - Yes, it's true that fixing this issue may require additional configuration changes in your deployment environments. However, it's important to address this issue as soon as possible to ensure that your Synapse Analytics workloads can continue to access the required secrets.

You may want to consider testing any configuration changes in a non-production environment before deploying them to your production environment to minimize the risk of any unintended consequences.

Overall, while addressing this issue may require some effort, it's important to ensure that your Synapse Analytics workloads can continue to function properly and securely.

Hope this helps. Do let us know if you any further queries.

Share via

Azure Synapse mssparkutils.credentials.getSecretWithLS started failing on parsing Key Vault URL.

0 additional answers

Your answer