Why Databricks cluster is creating more executors after all the job gets completed?

Question

Why Databricks cluster is creating more executors after all the job gets completed?

Manash 51

I am running a pyspark scripts where nearly 13 CSV files are read from azure data lake, they are then joined one by one, and the final file is written into the same data-lake but on to a different container as a parquet file. I repeated this process several times and noticed in the SparkUI every time that more executors are getting created few minutes after my jobs gets completed. No one else is using my cluster and no more jobs have been triggered after writing the final file. This looks strange to me.

Why more executors are getting allocated after all jobs are completed?

Spark

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-12-07T19:35:33.7+00:00

Hello Manash,

Welcome to the Microsoft Q&A forum.

It seems that the additional executors you are seeing in the SparkUI are part of the dynamic allocation feature in Apache Spark(executors are still being used by the Spark application)

Dynamic allocation is a feature in Spark that allows to automatically add or remove executors based on the workload. When a job is submitted to Spark, it will initially allocate a certain number of executors based on the configuration settings. However, if the workload increases, Spark can dynamically add more executors to handle the additional load. Similarly, if the workload decreases, Spark can remove executors to free up resources.

In your case, it seems Spark is adding more executors to handle any additional workload that may be present, even if there are no active jobs running. This is because Spark may not immediately release the executors after a job is completed, in case there is additional work to be done.

If you want to prevent Spark from adding more executors after your job is completed, you can try disabling dynamic allocation by setting the following configuration property in your Spark configuration:

spark.dynamicAllocation.enabled=false

This will disable dynamic allocation and prevent Spark from adding more executors after your job is completed.

Please note: If you manuay stop and start the cluster(spark application) then the executors will be released.

I hope this helps! Let me know if you have any further questions.

Reference document:
https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation
Manash 51 Reputation points

2023-12-08T02:56:56.65+00:00

Thanks for sharing these details. However as per this site the default value is false. When I tried to check the default value, I receive the below error.

I am running on Databricks runtime versions 14.0 with spark on version 3.5.0. How can I check the default value of this settings?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-12-08T21:05:14.5+00:00
Hello Manash,

spark.dynamicAllocation.enabled is a cluster-level property in Spark. This means that the setting applies to all Spark applications running on the cluster. If you set this property at the cluster level, it will be enabled for all sessions and jobs running on that cluster.

However, you can override this property for individual Spark sessions or jobs by setting it in your notebook or job configuration.

Cluster level:

Go to the compute section in your Databricks workspace.

Select the cluster and go to Configuration > Advanced Options.

Add spark.dynamicAllocation.enabled as a Spark property and set its value to true.

Spark Submit: You can pass the configuration as a command-line argument when you submit your Spark job.

Before SparkContext Initialization: If you’re writing a PySpark script, you can set this configuration before initializing the SparkContext

from pyspark import SparkConf, SparkContext conf = SparkConf().setAppName("app").set("spark.dynamicAllocation.enabled", "false") sc = SparkContext.getOrCreate(conf=conf)

I hope this helps.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-12-11T20:56:47.81+00:00

Hello Manash,

I am checking to see if you had a chance to look into the above response.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-12-16T00:02:31.7466667+00:00

Hello Manash,

I am checking to see if you had a chance to look into the above response.
Manash 51 Reputation points

2023-12-18T05:11:02.1833333+00:00
Sorry I was on vacation and thanks for sharing the details, I am able to set the same on cluster level now using init scripts.

spark.conf.get("spark.dynamicAllocation.enabled") 'true'

I tried to override the above settings on cluster level using notebook and it did not work. Am I doing anything wrong in changing the settings from notebook?

from pyspark import SparkConf, SparkContext conf = SparkConf().setAppName("app").set("spark.dynamicAllocation.enabled", "false") sc = SparkContext.getOrCreate(conf=conf) spark.conf.get("spark.dynamicAllocation.enabled") 'true'

Bhargava-MSFT 31,261 Microsoft Employee Moderator

Can you try with SparkSession object instead of the SparkConf

from pyspark.sql import SparkSession

# Create a SparkSession object
spark = SparkSession.builder.appName("app").config("spark.dynamicAllocation.enabled", "true").getOrCreate()

spark.conf.get("spark.dynamicAllocation.enabled")

Manash 51 Reputation points

2023-12-19T03:51:36.0266667+00:00

Thanks it worked!
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-12-19T18:44:32.9766667+00:00

Thank you, Manash
Manash 51 Reputation points

2023-12-21T03:31:20.35+00:00

@Bhargava-MSFT , I think I did not checked this clearly. I ran the same script for which I raised this question. Now my execution summary looks like below (all my jobs used only Executor 0 and 1 only)

Event timelines looks like below.

I have set the "spark.dynamicAllocation.enabled" parameter to true on cluster level and then again set this parameter to false using the script you provided.

What I see is that Executors 2 till 5 were created during the jobs running time but never used. This looks strange!

Only executor 0 & 1 were in use. Will there be charges incurred for executor 2 till 5?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-12-27T17:33:58.7066667+00:00

Hello Manash,

I am checking to see if you have any further questions here.
Manash 51 Reputation points

2023-12-28T11:21:36.3066667+00:00

Its all good now. Between I don't see a n option to accept what you said as an answer. How do I accept it?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-12-28T16:15:31.0066667+00:00

Hello Manash,

Thank you. I have converted my comment to answer. Please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.

Accepted answer

0 additional answers

Your answer

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-12-07T19:35:33.7+00:00

Hello Manash,

Welcome to the Microsoft Q&A forum.

It seems that the additional executors you are seeing in the SparkUI are part of the dynamic allocation feature in Apache Spark(executors are still being used by the Spark application)

Dynamic allocation is a feature in Spark that allows to automatically add or remove executors based on the workload. When a job is submitted to Spark, it will initially allocate a certain number of executors based on the configuration settings. However, if the workload increases, Spark can dynamically add more executors to handle the additional load. Similarly, if the workload decreases, Spark can remove executors to free up resources.

In your case, it seems Spark is adding more executors to handle any additional workload that may be present, even if there are no active jobs running. This is because Spark may not immediately release the executors after a job is completed, in case there is additional work to be done.

If you want to prevent Spark from adding more executors after your job is completed, you can try disabling dynamic allocation by setting the following configuration property in your Spark configuration:

spark.dynamicAllocation.enabled=false

This will disable dynamic allocation and prevent Spark from adding more executors after your job is completed.

Please note: If you manuay stop and start the cluster(spark application) then the executors will be released.

I hope this helps! Let me know if you have any further questions.

Reference document:
https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation
Manash 51 Reputation points

2023-12-08T02:56:56.65+00:00

Thanks for sharing these details. However as per this site the default value is false. When I tried to check the default value, I receive the below error.

I am running on Databricks runtime versions 14.0 with spark on version 3.5.0. How can I check the default value of this settings?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-12-08T21:05:14.5+00:00

Hello Manash,

spark.dynamicAllocation.enabled is a cluster-level property in Spark. This means that the setting applies to all Spark applications running on the cluster. If you set this property at the cluster level, it will be enabled for all sessions and jobs running on that cluster.

However, you can override this property for individual Spark sessions or jobs by setting it in your notebook or job configuration.

Cluster level:

Go to the compute section in your Databricks workspace.

Select the cluster and go to Configuration > Advanced Options.

Add spark.dynamicAllocation.enabled as a Spark property and set its value to true.

Spark Submit: You can pass the configuration as a command-line argument when you submit your Spark job.

Before SparkContext Initialization: If you’re writing a PySpark script, you can set this configuration before initializing the SparkContext

from pyspark import SparkConf, SparkContext conf = SparkConf().setAppName("app").set("spark.dynamicAllocation.enabled", "false") sc = SparkContext.getOrCreate(conf=conf)

I hope this helps.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-12-11T20:56:47.81+00:00

Hello Manash,

I am checking to see if you had a chance to look into the above response.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-12-16T00:02:31.7466667+00:00

Hello Manash,

I am checking to see if you had a chance to look into the above response.
Manash 51 Reputation points

2023-12-18T05:11:02.1833333+00:00

Sorry I was on vacation and thanks for sharing the details, I am able to set the same on cluster level now using init scripts.

spark.conf.get("spark.dynamicAllocation.enabled") 'true'

I tried to override the above settings on cluster level using notebook and it did not work. Am I doing anything wrong in changing the settings from notebook?

from pyspark import SparkConf, SparkContext conf = SparkConf().setAppName("app").set("spark.dynamicAllocation.enabled", "false") sc = SparkContext.getOrCreate(conf=conf) spark.conf.get("spark.dynamicAllocation.enabled") 'true'
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-12-18T22:55:57.22+00:00

Can you try with SparkSession object instead of the SparkConf

from pyspark.sql import SparkSession # Create a SparkSession object spark = SparkSession.builder.appName("app").config("spark.dynamicAllocation.enabled", "true").getOrCreate() spark.conf.get("spark.dynamicAllocation.enabled")
Manash 51 Reputation points

2023-12-19T03:51:36.0266667+00:00

Thanks it worked!
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-12-19T18:44:32.9766667+00:00

Thank you, Manash
Manash 51 Reputation points

2023-12-21T03:31:20.35+00:00

@Bhargava-MSFT , I think I did not checked this clearly. I ran the same script for which I raised this question. Now my execution summary looks like below (all my jobs used only Executor 0 and 1 only)

Event timelines looks like below.

I have set the "spark.dynamicAllocation.enabled" parameter to true on cluster level and then again set this parameter to false using the script you provided.

What I see is that Executors 2 till 5 were created during the jobs running time but never used. This looks strange!

Only executor 0 & 1 were in use. Will there be charges incurred for executor 2 till 5?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-12-27T17:33:58.7066667+00:00

Hello Manash,

I am checking to see if you have any further questions here.
Manash 51 Reputation points

2023-12-28T11:21:36.3066667+00:00

Its all good now. Between I don't see a n option to accept what you said as an answer. How do I accept it?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-12-28T16:15:31.0066667+00:00

Hello Manash,

Thank you. I have converted my comment to answer. Please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.

Answer 1

Hello Manash,

When dynamic allocation enabled, spark will possibly acquire much more executors than expected. This means that your application may give resources back to the cluster if they are no longer used and request them again later when there is demand.

My understanding is, If the additional executors were created but not used during the job execution, then you will not be charged for those executors.

one another possible reason for the additional executors could be due to garbage collection. When Spark runs a job, it creates a number of objects in memory. These objects are managed by the Java Virtual Machine and are periodically cleaned up by the garbage collector. If the garbage collector is not able to keep up with the rate of object creation, it may cause the JVM to run out of memory. To prevent this, Spark may add additional executors to the pool to handle the increased workload.

Please refer the below document:
https://www.databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html

Share via

Why Databricks cluster is creating more executors after all the job gets completed?

0 additional answers

Your answer