AzureML Notebook Spark Serverless Not Working

Cameron Grimshaw-Jones 11 Reputation points
2023-05-14T11:07:39.1133333+00:00

Having issues trying to run Spark on my notebook. The number of executors I'm allowed seems to always stay negative and changes randomly.

User's image

User's image

Getting the error message below when I apply these settings

User's image

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,539 questions
0 comments No comments
{count} votes

3 answers

Sort by: Most helpful
  1. Konstantinos Passadis 17,126 Reputation points
    2023-05-14T11:56:44.13+00:00

    Hello @Cameron Grimshaw-Jones !

    Welcome to Microsoft QnA!

    i see that your AzureML Notebooks are producing errors with negative values on the executors

    Can you kinldy post your code ? Please redact sensitive information

    I can see the line

    spark.dynamicAllocation.enabled": "true" ( You might want to check this line also in the notebook)

    has also -1 so i suspect a misconfiguration

    We could offer more help with more details if possible

    I hope this helps!

    Kindly mark the answer as Accepted and Upvote in case it helped!

    Regards

    0 comments No comments

  2. Cameron Grimshaw-Jones 11 Reputation points
    2023-05-14T12:02:43.6933333+00:00

    Hey,

    So it fails in the first code block. I'm only doing a few imports at this bit

    User's image

    User's image

    It also doesn't let me update any of the options to apply them

    0 comments No comments

  3. Konstantinos Passadis 17,126 Reputation points
    2023-05-14T12:23:24.5166667+00:00

    Hello @Cameron Grimshaw-Jones !

    Thanks for the info

    I can guide you to these links especially here:

    https://learn.microsoft.com/en-us/azure/machine-learning/apache-spark-azure-ml-concepts?view=azureml-api-2

    https://learn.microsoft.com/en-us/azure/machine-learning/how-to-submit-spark-jobs?view=azureml-api-2&tabs=cli

    And Here:

    https://learn.microsoft.com/en-us/azure/machine-learning/apache-spark-environment-configuration?view=azureml-api-2

    Please go through the documentation carefully and see that you follow prerequisites

    I hope this helps!

    Kindly mark the answer as Accepted and Upvote in case it helped!

    Regards