How to reduce unnecessary high memory usage in a Databricks cluster?

Question

How to reduce unnecessary high memory usage in a Databricks cluster?

Senad Hadzikic 20

We are having unnecessary high memory usage even when nothing is running on the cluster. When the cluster first starts, it's fine, but when I run a script and it finishes executing, nothing gets back to the idle (initial) state (even hours after nothing else was executed).

Screenshot 2024-05-08 at 10.53.08

Cluster config:
Screenshot 2024-05-08 at 10.56.09

Some settings i tried:
Screenshot 2024-05-08 at 10.56.41

Spark Config:
spark.executor.extraJavaOptions -XX:+UseG1GC -XX:MaxGCPauseMillis=500 -XX:ParallelGCThreads=20 -XX:ConcGCThreads=5 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:G1HeapRegionSize=8M spark.memory.storageFraction 0.5 spark.dynamicAllocation.maxExecutors 10 spark.driver.extraJavaOptions -XX:+UseG1GC -XX:MaxGCPauseMillis=500 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=10M -Xloggc:/databricks/driver/logs/gc.log -XX:G1HeapRegionSize=8M -XX:+ExplicitGCInvokesConcurrent spark.dynamicAllocation.enabled true spark.memory.fraction 0.6 spark.dynamicAllocation.minExecutors 1

PRADEEPCHEEKATLA 91,861 Reputation points

2024-05-08T11:01:48.0733333+00:00
@Senad Hadzikic - Thanks for the question and using MS Q&A platform.

Based on the provided information, it seems that the cluster is not getting back to the idle state even after the script execution is completed. To reduce unnecessary high memory usage in a Databricks cluster, you can try the following steps:

Turn on Auto Optimize by adding the following properties to your Spark configuration:
spark.databricks.delta.optimizeWrite.enabled true spark.databricks.delta.autoCompact.enabled true

Configure your cluster depending on your integration and scaling needs. For cluster configuration details, see Configure clusters.

You can also try setting the spark.dynamicAllocation.enabled property to true to enable dynamic allocation of executors. This will allow the cluster to automatically scale up or down based on the workload.

Additionally, you can try setting the spark.memory.fraction property to a lower value, such as 0.4 or 0.5, to reduce the amount of memory allocated to Spark.

You can also try setting the spark.dynamicAllocation.minExecutors property to a lower value, such as 1, to reduce the number of executors that are running when the cluster is idle.

If these steps do not help, you may need to further investigate the root cause of the high memory usage. You can use the Databricks monitoring and logging features to identify the source of the memory usage and take appropriate actions to optimize your cluster.

Hope this helps. Do let us know if you any further queries.

Senad Hadzikic 20

Hello and thank you for the answer, I'll try that out.
The monitoring tools didn't help much in debugging the issue to be honest. I've done some of my own investigations:

Ran this notebook:

%sh
free -h
top -b -n 1 | head -n 20
jps -l

The result:

               total        used        free      shared  buff/cache   available
Mem:            57Gi        35Gi        12Gi       0.0Ki       9.5Gi        21Gi
Swap:            9Gi       0.0Ki         9Gi
top - 11:45:55 up 15 min,  0 users,  load average: 1.38, 1.65, 1.44
Tasks:  32 total,   1 running,  30 sleeping,   0 stopped,   1 zombie
%Cpu(s):  8.2 us,  0.4 sy,  0.0 ni, 91.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  58770.0 total,  12420.7 free,  36636.6 used,   9712.7 buff/cache
MiB Swap:  10240.0 total,  10240.0 free,      0.0 used.  22133.4 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    759 root      20   0   57.3g  33.3g 186028 S 126.7  58.0  17:50.24 java
    365 root      20   0 8775452   1.0g  47792 S   6.7   1.8   1:06.52 java
      1 root      20   0   17964  10776   8272 S   0.0   0.0   0:00.19 systemd
     45 root      20   0   39572  11828  10740 S   0.0   0.0   0:00.05 systemd+
     55 systemd+  20   0   25540  12980   8788 S   0.0   0.0   0:00.01 systemd+
     57 root      20   0    3888   2600   2360 S   0.0   0.0   0:00.00 cron
     58 message+  20   0    8592   4776   4192 S   0.0   0.0   0:00.05 dbus-da+
     61 root      20   0   35280  22048  10344 S   0.0   0.0   0:00.16 network+
     63 syslog    20   0  160960   5056   4496 S   0.0   0.0   0:00.01 rsyslogd
     67 root      20   0   15360   7616   6608 S   0.0   0.0   0:00.02 systemd+
     72 root      20   0   15436   8808   7256 S   0.0   0.0   0:00.00 sshd
     74 root      20   0    2820   1108   1020 S   0.0   0.0   0:00.00 agetty
     76 root      20   0  121200  23328  12720 S   0.0   0.0   0:00.14 unatten+
759 com.databricks.backend.daemon.driver.DriverDaemon
508 org.apache.spark.deploy.master.Master
365 com.databricks.spark.chauffeur.ChauffeurDaemon
5598 sun.tools.jps.Jps

Then this notebook:

%python
import os


pid = os.popen("jps | grep DriverDaemon | awk '{print $1}'").read().strip()
heap_dump_command = f"jmap -dump:live,format=b,file=/tmp/heapdump_{pid}.hprof {pid}"
print(os.popen(heap_dump_command).read())


gc_stats_command = f"jstat -gc {pid}"
print(os.popen(gc_stats_command).read())


memory_summary_command = f"jmap -heap {pid}"
print(os.popen(memory_summary_command).read())


open_files_command = "lsof | grep java | wc -l"
print(os.popen(open_files_command).read())

The result:

1.
Dumping heap to /tmp/heapdump_759.hprof ...
Heap dump file created

2.
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT   
 0.0    0.0    0.0    0.0   29671424.0 73728.0  17424384.0  566112.3  499948.0 474495.4  0.0    0.0       76    4.449   1      0.958    5.406

3.
Attaching to process ID 759, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.392-b08

using thread-local object allocation.
Garbage-First (G1) GC with 13 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 48226107392 (45992.0MB)
   NewSize                  = 1363144 (1.2999954223632812MB)
   MaxNewSize               = 28932308992 (27592.0MB)
   OldSize                  = 5452592 (5.1999969482421875MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 8388608 (8.0MB)

Heap Usage:
G1 Heap:
   regions  = 5749
   capacity = 48226107392 (45992.0MB)
   used     = 1418559840 (1352.8440856933594MB)
   free     = 46807547552 (44639.15591430664MB)
   2.9414769648925017% used
G1 Young Generation:
Eden Space:
   regions  = 100
   capacity = 30383538176 (28976.0MB)
   used     = 838860800 (800.0MB)
   free     = 29544677376 (28176.0MB)
   2.7609055770292654% used
Survivor Space:
   regions  = 0
   capacity = 0 (0.0MB)
   used     = 0 (0.0MB)
   free     = 0 (0.0MB)
   0.0% used
G1 Old Generation:
   regions  = 70
   capacity = 17842569216 (17016.0MB)
   used     = 579699040 (552.8440856933594MB)
   free     = 17262870176 (16463.15591430664MB)
   3.2489661829652055% used

91848 interned Strings occupying 12598088 bytes.

4.
lsof: WARNING: can't stat() fuse file system /local_disk0/.wsfs
      Output information may be incomplete.

If this helps you give even better feedback, it didn't help me though.

Senad Hadzikic 20

Thank you for the answer. I applied the suggested changes.
The memory usage is still really huge: Screenshot 2024-05-08 at 15.00.11

I ran this notebook to get little more details:

%sh
free -h
top -b -n 1 | head -n 20
jps -l

The result:

               total        used        free      shared  buff/cache   available
Mem:            57Gi        35Gi        10Gi       0.0Ki        10Gi        21Gi
Swap:            9Gi       0.0Ki         9Gi
top - 12:54:50 up 32 min,  0 users,  load average: 2.48, 1.54, 1.23
Tasks:  30 total,   1 running,  28 sleeping,   0 stopped,   1 zombie
%Cpu(s):  6.5 us,  0.0 sy,  0.0 ni, 93.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  58770.0 total,  11114.9 free,  36754.9 used,  10900.2 buff/cache
MiB Swap:  10240.0 total,  10240.0 free,      0.0 used.  22015.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    796 root      20   0   57.2g  33.7g 185796 S 100.0  58.8  26:35.27 java
    389 root      20   0 8704676   1.0g  47760 S   6.2   1.8   1:27.04 java
      1 root      20   0   17972  10808   8284 S   0.0   0.0   0:00.23 systemd
     41 root      20   0   39576  11764  10668 S   0.0   0.0   0:00.06 systemd+
     71 systemd+  20   0   25540  12948   8756 S   0.0   0.0   0:00.01 systemd+
     77 root      20   0    3888   2396   2160 S   0.0   0.0   0:00.00 cron
     78 message+  20   0    8592   4764   4180 S   0.0   0.0   0:00.04 dbus-da+
     81 root      20   0   35228  21864  10208 S   0.0   0.0   0:00.16 network+
     83 syslog    20   0  160960   4924   4364 S   0.0   0.0   0:00.01 rsyslogd
     87 root      20   0   15512   7772   6780 S   0.0   0.0   0:00.03 systemd+
     92 root      20   0   15436   8880   7332 S   0.0   0.0   0:00.00 sshd
     94 root      20   0    2820   1072    984 S   0.0   0.0   0:00.00 agetty
     99 root      20   0  121080  22996  12532 S   0.0   0.0   0:00.12 unatten+
6881 sun.tools.jps.Jps
532 org.apache.spark.deploy.master.Master
389 com.databricks.spark.chauffeur.ChauffeurDaemon
796 com.databricks.backend.daemon.driver.DriverDaemon

This is happening on both of our clusters that we use.

PRADEEPCHEEKATLA 91,861 Reputation points

2024-05-09T05:21:33.6966667+00:00
@Senad Hadzikic - Based on the output you provided, it looks like your cluster is still using a significant amount of memory even after applying the suggested changes. Here are a few more things you can try:

Check if there are any long-running Spark jobs or notebooks that might be consuming memory. You can check this by going to the "Jobs" and "Notebooks" tabs in the Databricks workspace and looking for any jobs or notebooks that have been running for a long time.

Try reducing the spark.executor.memory value to allocate less memory to each executor. You can also try reducing the spark.driver.memory value to allocate less memory to the driver.

Check if there are any external libraries or dependencies that might be causing memory leaks. You can try removing any unnecessary libraries or dependencies and see if that helps reduce memory usage.

Try monitoring the cluster's memory usage over time using the Databricks monitoring tools. This can help you identify any patterns or trends in memory usage and adjust your settings accordingly.

Finally, you can try optimizing your Spark code to reduce memory usage. This can include techniques such as using smaller data partitions, using more efficient data serialization formats, and optimizing your Spark configuration settings.

I hope this helps! Let me know if you have any further questions.
Senad Hadzikic 20 Reputation points

2024-05-09T11:14:59.3033333+00:00

Thank you for the response once again. I have optimized the code to the max, as much as possible. These tips helped a bit but we still have a really huge memory usage. I was wondering if there's an additional possible way to release the cached memory which is 22GB (that is not restarting the cluster itself)? I've tried a couple of steps through a notebook cell but still no luck.

And this is the OS-level cache. I just want to be able to clear it, but can't manage to do it with a notebook it seems. Any other way?
PRADEEPCHEEKATLA 91,861 Reputation points

2024-05-14T04:09:06.0466667+00:00

@Senad Hadzikic - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

7 answers

Your answer

PRADEEPCHEEKATLA 91,861 Reputation points

2024-05-08T11:01:48.0733333+00:00

@Senad Hadzikic - Thanks for the question and using MS Q&A platform.

Based on the provided information, it seems that the cluster is not getting back to the idle state even after the script execution is completed. To reduce unnecessary high memory usage in a Databricks cluster, you can try the following steps:

Turn on Auto Optimize by adding the following properties to your Spark configuration:
spark.databricks.delta.optimizeWrite.enabled true spark.databricks.delta.autoCompact.enabled true

Configure your cluster depending on your integration and scaling needs. For cluster configuration details, see Configure clusters.

You can also try setting the spark.dynamicAllocation.enabled property to true to enable dynamic allocation of executors. This will allow the cluster to automatically scale up or down based on the workload.

Additionally, you can try setting the spark.memory.fraction property to a lower value, such as 0.4 or 0.5, to reduce the amount of memory allocated to Spark.

You can also try setting the spark.dynamicAllocation.minExecutors property to a lower value, such as 1, to reduce the number of executors that are running when the cluster is idle.

If these steps do not help, you may need to further investigate the root cause of the high memory usage. You can use the Databricks monitoring and logging features to identify the source of the memory usage and take appropriate actions to optimize your cluster.

Hope this helps. Do let us know if you any further queries.
Senad Hadzikic 20 Reputation points

2024-05-08T13:04:43.77+00:00

Thank you for the answer. I applied the suggested changes.
The memory usage is still really huge:

I ran this notebook to get little more details:

%sh free -h top -b -n 1 | head -n 20 jps -l

The result:

total used free shared buff/cache available Mem: 57Gi 35Gi 10Gi 0.0Ki 10Gi 21Gi Swap: 9Gi 0.0Ki 9Gi top - 12:54:50 up 32 min, 0 users, load average: 2.48, 1.54, 1.23 Tasks: 30 total, 1 running, 28 sleeping, 0 stopped, 1 zombie %Cpu(s): 6.5 us, 0.0 sy, 0.0 ni, 93.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 58770.0 total, 11114.9 free, 36754.9 used, 10900.2 buff/cache MiB Swap: 10240.0 total, 10240.0 free, 0.0 used. 22015.1 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 796 root 20 0 57.2g 33.7g 185796 S 100.0 58.8 26:35.27 java 389 root 20 0 8704676 1.0g 47760 S 6.2 1.8 1:27.04 java 1 root 20 0 17972 10808 8284 S 0.0 0.0 0:00.23 systemd 41 root 20 0 39576 11764 10668 S 0.0 0.0 0:00.06 systemd+ 71 systemd+ 20 0 25540 12948 8756 S 0.0 0.0 0:00.01 systemd+ 77 root 20 0 3888 2396 2160 S 0.0 0.0 0:00.00 cron 78 message+ 20 0 8592 4764 4180 S 0.0 0.0 0:00.04 dbus-da+ 81 root 20 0 35228 21864 10208 S 0.0 0.0 0:00.16 network+ 83 syslog 20 0 160960 4924 4364 S 0.0 0.0 0:00.01 rsyslogd 87 root 20 0 15512 7772 6780 S 0.0 0.0 0:00.03 systemd+ 92 root 20 0 15436 8880 7332 S 0.0 0.0 0:00.00 sshd 94 root 20 0 2820 1072 984 S 0.0 0.0 0:00.00 agetty 99 root 20 0 121080 22996 12532 S 0.0 0.0 0:00.12 unatten+ 6881 sun.tools.jps.Jps 532 org.apache.spark.deploy.master.Master 389 com.databricks.spark.chauffeur.ChauffeurDaemon 796 com.databricks.backend.daemon.driver.DriverDaemon

This is happening on both of our clusters that we use.
PRADEEPCHEEKATLA 91,861 Reputation points

2024-05-09T05:21:33.6966667+00:00

@Senad Hadzikic - Based on the output you provided, it looks like your cluster is still using a significant amount of memory even after applying the suggested changes. Here are a few more things you can try:

Check if there are any long-running Spark jobs or notebooks that might be consuming memory. You can check this by going to the "Jobs" and "Notebooks" tabs in the Databricks workspace and looking for any jobs or notebooks that have been running for a long time.

Try reducing the spark.executor.memory value to allocate less memory to each executor. You can also try reducing the spark.driver.memory value to allocate less memory to the driver.

Check if there are any external libraries or dependencies that might be causing memory leaks. You can try removing any unnecessary libraries or dependencies and see if that helps reduce memory usage.

Try monitoring the cluster's memory usage over time using the Databricks monitoring tools. This can help you identify any patterns or trends in memory usage and adjust your settings accordingly.

Finally, you can try optimizing your Spark code to reduce memory usage. This can include techniques such as using smaller data partitions, using more efficient data serialization formats, and optimizing your Spark configuration settings.

I hope this helps! Let me know if you have any further questions.
Senad Hadzikic 20 Reputation points

2024-05-09T11:14:59.3033333+00:00

Thank you for the response once again. I have optimized the code to the max, as much as possible. These tips helped a bit but we still have a really huge memory usage. I was wondering if there's an additional possible way to release the cached memory which is 22GB (that is not restarting the cluster itself)? I've tried a couple of steps through a notebook cell but still no luck.

And this is the OS-level cache. I just want to be able to clear it, but can't manage to do it with a notebook it seems. Any other way?
PRADEEPCHEEKATLA 91,861 Reputation points

2024-05-14T04:09:06.0466667+00:00

@Senad Hadzikic - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Luigi Greselin 5

I am experiencing the same problem. I even switched from a 28 Gb Memory cluster to a 56 Gb. It gets completely full after the first run

0 comments

Answer 2

Ben Gislason 5

I have a similar problem and it does not seem like the root question at hand is getting answered. Why is there so much memory in the first place? I am experiencing exact same situation with a very simple join operation on a 3x9 pyspark dataframe. As soon as I click run at all, same memory consumption as shown above is seen in mine. Where is all this memory usage coming from? My same code ran fine a week ago.

0 comments

Answer 3

@Senad Hadzikic - If you want to release the cached memory in your Databricks cluster without restarting the cluster itself, you can try the following steps:

Use the spark.catalog.clearCache() method to clear the cached data in Spark. This method removes all cached data from memory and disk. You can run this method in a notebook cell to clear the cached data.
Use the dbutils.fs.unmount() method to unmount any mounted file systems. Mounted file systems can consume memory, so unmounting them can help free up memory. You can run this method in a notebook cell to unmount any mounted file systems.
Use the sync command to flush the file system buffers and free up memory. You can run this command in a notebook cell to flush the file system buffers.
Use the echo /proc/sys/vm/drop_caches command to drop the page cache, dentries, and inodes. This command can help free up memory that is being used by the operating system cache. However, this command requires root access, so you might need to contact your Databricks administrator to run this command.
Consider using a different type of Databricks cluster. For example, you might try using a different instance type or a different number of nodes to see if this improves memory usage.

Note that these steps might not free up all of the memory that is being used by your Databricks cluster, but they can help free up some memory. If you are still experiencing high memory usage after trying these steps, you might need to consider opening a support ticket for further assistance.

Hope this helps. Do let us know if you any further queries.

Alex 0 Reputation points

2024-07-08T17:44:30.65+00:00

None of this helps. spark.catalog.clearCache() does absolutely nothing. If this issue has been addressed somewhere else, please point us to the resource, otherwise, please look into this.

Answer 4

Alex 0

Hi @PRADEEPCHEEKATLA , anybody at Microsoft looking into this? If not, can you point us to the right Databricks forum/contact. It's been months without a solution.

0 comments

Answer 5

Alex 0

Same problem here, I did the same as the user in the last comment, increased the cluster memory. But it still shows almost 100% memory utilization long after the script in a Notebook has completed. Ironically, clearing the cache does not help. Only way to solve it is to restart the cluster.

User's image

0 comments

Share via

How to reduce unnecessary high memory usage in a Databricks cluster?

7 answers

Your answer