Spark_Ambiguous_Executor_MaxExecutorFailures

Seelan 0 Reputation points
2024-05-06T08:06:10.67+00:00

Hi

I'm running a scheduled multiple run notebook using the below configuration, but I keep getting the below error

DAG = {

"activities": [

    {

        "name": "Notebook",

        "path": "Notebook1", 

        "timeoutPerCellInSeconds": 1800,

    },

    {

        "name": "Notebook2",

        "path": "Notebook2",

        "timeoutPerCellInSeconds": 1800

    },

    {

        "name": "Notebook3",

        "path": "Notebook3",

        "timeoutPerCellInSeconds": 1800

    },

     {

        "name": "Notebook4",

        "path": "Notebook4",

        "timeoutPerCellInSeconds": 1800

    },

     {

        "name": "Notebook5",

        "path": "Notebook5",

        "timeoutPerCellInSeconds": 1800

    },

     {

        "name": "Notebook6",

        "path": "Notebook6",

        "timeoutPerCellInSeconds": 1800

    },

     {

        "name": "Notebook7",

        "path": "Notebook7",

        "timeoutPerCellInSeconds": 1800

    },

],

"timeoutInSeconds": 43200,

"concurrency": 5

}

mssparkutils.notebook.runMultiple(DAG, {"displayDAGViaGraphviz": False})

Error :

Spark_Ambiguous_Executor_MaxExecutorFailures: Livy session has failed. Session state: Dead. Error code: Spark_Ambiguous_Executor_MaxExecutorFailures. Job failed during run time with state=[dead].

TSG:Application failed because too many executors failed. The number of acceptable

executor failures is controlled by the config spark.yarn.max.executor.failures.

To investigate this failure, look at the executor logs and error codes.

Source: Unknown.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,960 questions
{count} votes

1 answer

Sort by: Most helpful
  1. BhargavaGunnam-MSFT 27,326 Reputation points Microsoft Employee
    2024-05-10T23:04:05.6666667+00:00

    Hello Seelan,

    The task failed because the executor that is running was lost.

    This may happen because the task crashed the JVM.

    Here are the Possible causes:

    1. Driver memory issues
    2. Executor Memory issues
    3. Executor lost

    Please check executor logs and error codes and see if you find any more details.

    0 comments No comments