mssparkutils.credentials.getSecret() timeout errors

Question

I am running into an error that appears to randomly occur in my spark notebooks for Azure Synapse.

Here is the error I am getting:

Py4JJavaError: An error occurred while calling z:mssparkutils.credentials.getSecret.
: com.twitter.util.TimeoutException: 1.minutes
	at com.twitter.util.Future.$anonfun$within$1(Future.scala:1642)
	at com.twitter.util.Future$$anon$4.apply$mcV$sp(Future.scala:1693)
	at com.twitter.util.Monitor.apply(Monitor.scala:46)
	at com.twitter.util.Monitor.apply$(Monitor.scala:41)
	at com.twitter.util.NullMonitor$.apply(Monitor.scala:229)
	at com.twitter.util.Timer.$anonfun$schedule$2(Timer.scala:39)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.twitter.util.Local$.let(Local.scala:4904)
	at com.twitter.util.Timer.$anonfun$schedule$1(Timer.scala:39)
	at com.twitter.util.Monitor.apply(Monitor.scala:46)
	at com.twitter.util.Monitor.apply$(Monitor.scala:41)
	at com.twitter.util.NullMonitor$.apply(Monitor.scala:229)
	at com.twitter.util.Timer.$anonfun$schedule$2(Timer.scala:39)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.twitter.util.Local$.let(Local.scala:4904)
	at com.twitter.util.Timer.$anonfun$schedule$1(Timer.scala:39)
	at com.twitter.util.JavaTimer.$anonfun$scheduleOnce$1(Timer.scala:233)
	at com.twitter.util.JavaTimer$$anon$3.run(Timer.scala:264)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)

I have 10+ notebooks running and some days no notebooks have issues and other days a random 1 out of 10 notebooks will throw this error. I have a python function that is used in all of these notebooks that takes a the secret name, calls mssparkutils.credentials.getSecret(), and returns the secret value.

The notebooks that fail run fine on other days and I am having a hard time tracking down what is causing these intermittent failures to occur. These failures only occur when the spark notebook is called in a pipeline not running them manually.

Answer 1

@Richards, Sam (DG-STL-HQ)

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Error Message:

Py4JJavaError: An error occurred while calling z:mssparkutils.credentials.getSecret.
: com.twitter.util.TimeoutException: 1.minutes
	at com.twitter.util.Future.$anonfun$within$1(Future.scala:1642)
	at com.twitter.util.Future$$anon$4.apply$mcV$sp(Future.scala:1693)
	at com.twitter.util.Monitor.apply(Monitor.scala:46)
	at com.twitter.util.Monitor.apply$(Monitor.scala:41)
	at com.twitter.util.NullMonitor$.apply(Monitor.scala:229)
	at com.twitter.util.Timer.$anonfun$schedule$2(Timer.scala:39)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.twitter.util.Local$.let(Local.scala:4904)
	at com.twitter.util.Timer.$anonfun$schedule$1(Timer.scala:39)
	at com.twitter.util.Monitor.apply(Monitor.scala:46)
	at com.twitter.util.Monitor.apply$(Monitor.scala:41)
	at com.twitter.util.NullMonitor$.apply(Monitor.scala:229)
	at com.twitter.util.Timer.$anonfun$schedule$2(Timer.scala:39)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.twitter.util.Local$.let(Local.scala:4904)
	at com.twitter.util.Timer.$anonfun$schedule$1(Timer.scala:39)
	at com.twitter.util.JavaTimer.$anonfun$scheduleOnce$1(Timer.scala:233)
	at com.twitter.util.JavaTimer$$anon$3.run(Timer.scala:264)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)

Issue:

• Out of multiple notebooks running in Synapse Spark, some days no notebooks have issues and other days a random 1 out of 10 notebooks will throw this error. I have a python function that is used in all of these notebooks that takes a the secret name, calls mssparkutils.credentials.getSecret(), and returns the secret value.

The notebooks that fail run fine on other days and I am having a hard time tracking down what is causing these intermittent failures to occur. These failures only occur when the spark notebook is called in a pipeline not running them manually.

Solution:

• This could be because the number of Spark executors were not enough to handle the workloads and hence results in timeout issue or the other. In order to overcome the error, we can adjust Spark settings to dynamically allocate executors based on the workloads and then applied that in the pipelines that call the spark notebook. Below is on how to set dynamic allocation of executors on Spark pool configuration settings.

• If you would like to enable it on a per job basis, set the following in each of the notebooks:

conf.set("spark.dynamicAllocation.enabled","true")

For more info please refer to this article: Spark Configuration - Dynamic Allocation

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.