AzureML Notebook Spark Serverless Not Working

Question

Having issues trying to run Spark on my notebook. The number of executors I'm allowed seems to always stay negative and changes randomly.

Getting the error message below when I apply these settings

Answer 1

Hello @Cameron Grimshaw-Jones !

Welcome to Microsoft QnA!

i see that your AzureML Notebooks are producing errors with negative values on the executors

Can you kinldy post your code ? Please redact sensitive information

I can see the line

spark.dynamicAllocation.enabled": "true" ( You might want to check this line also in the notebook)

has also -1 so i suspect a misconfiguration

We could offer more help with more details if possible

I hope this helps!

Kindly mark the answer as Accepted and Upvote in case it helped!

Regards

Answer 2

Hey,

So it fails in the first code block. I'm only doing a few imports at this bit

It also doesn't let me update any of the options to apply them

Answer 3

Hello @Cameron Grimshaw-Jones !

Thanks for the info

I can guide you to these links especially here:

https://learn.microsoft.com/en-us/azure/machine-learning/apache-spark-azure-ml-concepts?view=azureml-api-2

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-submit-spark-jobs?view=azureml-api-2&tabs=cli

And Here:

https://learn.microsoft.com/en-us/azure/machine-learning/apache-spark-environment-configuration?view=azureml-api-2

Please go through the documentation carefully and see that you follow prerequisites

I hope this helps!

Kindly mark the answer as Accepted and Upvote in case it helped!

Regards