When using the %debug feature in databricks notebook, if the user forgets to cancel this command manually then it will continue to run until it is noticed or the cluster is manually terminated

Question

When using the %debug feature in databricks notebook, if the user forgets to cancel this command manually then it will continue to run until it is noticed or the cluster is manually terminated

Ashwini Gaikwad 130

Hello Team,

When using the %debug feature in azure databricks notebook, if the user forgets to cancel the command manually then it will continue to run for more than half hour or 3 days until it is noticed or the cluster is manually terminated. This behaviour leads to additional costs for the cluster.

Request you to please assist in this case, how this feature is auto-terminated after 30 mins let's say without manual intervention or are there any alternatives for it?

User's image

Regards,

Ashwini Gaikwad

J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-05-14T09:44:42.0266667+00:00

Hi @Ashwini Gaikwad
To end the debugging session, click at the upper-left of the cell or click at the top of the notebook. The debug session also automatically terminates after 30 minutes if the user does not use the debugging toolbar or debug console.
https://learn.microsoft.com/en-us/azure/databricks/notebooks/debugger

This behavior helps prevent long-running debugger sessions that could lead to unnecessary cluster resource usage and associated costs.

If you have observed instances where debugger sessions persist beyond the expected 30-minute timeout, it may indicate an issue with the auto-termination mechanism. In such cases, consider the following steps:

You can also set auto termination for a compute. During compute creation, you can specify an inactivity period in minutes after which you want the compute to terminate.

If the difference between the current time and the last command run on the compute is more than the inactivity period specified, Azure Databricks automatically terminates that compute.
https://learn.microsoft.com/en-us/azure/databricks/compute/clusters-manage

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.
Ashwini Gaikwad 130 Reputation points

2025-05-14T11:05:02.3633333+00:00

Hello @J N S S Kasyap thank you for your response.

On cluster there is already policy set which terminates the cluster after 45 mins of inactivity. But when %debug command is ran, the cluster doesn't stop at 45 mins or auto-terminates, you can see in the diagram below

First diagram shows the cluster is running for 3 days even though auto-termination for cluster is set to 45 mins

Second diagram states the cluster is running for more than 30 mins

Can you suggest what can be done here to auto-terminate the debug session after 30 mins?

Regards,

Ashwini
J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-05-14T14:40:14.56+00:00

@Ashwini Gaikwad
The %debug command initiates an interactive debugging session in Azure Databricks, which may be considered an active process by the cluster, preventing it from being marked as "inactive" for the purposes of auto-termination.
The auto-termination feature in Azure Databricks monitors activity such as Spark jobs but certain operations, including user-defined local processes or debugging sessions, may not be properly recognized as inactive, even if no explicit commands are running. This is particularly true in older Spark versions or specific runtime environments where activity reporting may have limitations.

Please consider the below options:

Older Spark or Databricks Runtime versions may have limitations in accurately reporting cluster activity, leading to premature or delayed termination during debugging sessions. Upgrade to the latest Databricks Runtime version (e.g., 12.2 LTS or higher, as of your context) to benefit from bug fixes and improvements in auto-termination logic.

The %debug session is not auto-terminating after 30 minutes, you can programmatically terminate it using the Databricks Clusters API to avoid manual intervention.Create a script or job to check for long-running clusters and terminate them after a set period.

If the %debug session is keeping the cluster alive, you can periodically run a lightweight command to reset the inactivity timer, ensuring the cluster recognizes activity and adheres to the auto-termination policy. Schedule a lightweight Spark job (e.g., spark.sql("SELECT 1").collect()) to run at regular intervals every 10–15 minutes during the debugging session

https://community.databricks.com/t5/data-engineering/how-to-prevent-my-cluster-to-shut-down-after-inactivity/td-p/29608

I hope this information helps. Please do let us know if you have any further queries.
J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-05-15T07:53:26.4133333+00:00

@Ashwini Gaikwad
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Ashwini Gaikwad 130 Reputation points

2025-05-15T15:08:03.89+00:00

@J N S S Kasyap all our clusters are running 14.3 or 15.4 LTS so we are using higher databricks runtime versions.

Can you let me know how can I stop the %debug feature after 30 mins approx so that it doesn't run for 3 days or so.

Can you share the command or a script or a notebook to auto-terminate the %debug activity or the cluster?

Regards,

Ashwini Gaikwad
Singireddi JagadishKumar 0 Reputation points Microsoft External Staff Moderator

2025-05-16T12:06:51.99+00:00

Hi Ashwini Gaikwad

Use Spark Watchdog Timeout

Databricks provides a Query Watchdog feature that can help terminate long-running commands, including %debug sessions, though it’s not officially documented for %debug. It’s worth testing in your environment.

Cluster-Level Configuration
Navigate to the Compute in your Databricks workspace.
Select your cluster and click Edit. Under Advanced Options > Spark Config, add:

spark.databricks.queryWatchdog.enabled= true

spark.databricks.queryWatchdog.timeout = 1800

This sets a 30-minute timeout (1800 seconds) for any command, including %debug.

Click Confirm and Restart the cluster.

Singireddi JagadishKumar 0 Microsoft External Staff Moderator

Hi @Ashwini Gaikwad

If you need a more customizable debug session with timeout control, we recommend using the approach shown in Command #2.

In this method, the %debug magic command is encapsulated within a function that accepts a user-defined timeout duration (in seconds). The function will automatically terminate the debug session once the specified timeout is reached, helping to avoid unintended cluster usage and associated costs.

Therefore, instead of directly using the %debug command, please use the code provided in Command #2. This ensures that the debug session will remain active only for the defined duration, after which it will be automatically terminated.

For eg, you may run the following two commands for better understanding on how it works,

#Command 1:

def divide(a, b):
return a / b
result = divide(10, 0)  # This raises ZeroDivisionError

#Command 2:

import time
import threading
import os
import signal
# Set the timeout duration (in seconds, e.g., 1800 seconds = 30 minutes)
TIMEOUT_DURATION = 30
def terminate_debug_session():
"""Terminate the debug session by sending a termination signal."""
print("Timeout reached. Terminating the debug session and running commands...")
# Get the current process ID
pid = os.getpid()
# Send a SIGINT signal to the current process to interrupt the debug session
os.kill(pid, signal.SIGINT)
# Optionally, exit the notebook to prevent further execution
dbutils.notebook.exit("Debug session terminated due to timeout.")
# Start the timer in a separate thread
timer_thread = threading.Timer(TIMEOUT_DURATION, terminate_debug_session)
timer_thread.start()
# Start debugging
%debug
# Cancel the timer if the debug session is completed manually
timer_thread.cancel()

Singireddi JagadishKumar 0 Reputation points Microsoft External Staff Moderator

2025-05-20T04:33:34.1166667+00:00

Hi Ashwini Gaikwad
I wanted to follow up to check if you had an opportunity to review my response to your query. Please let us know if it addressed your concerns, and feel free to reach out with any additional questions or clarifications.

Your answer

J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-05-14T09:44:42.0266667+00:00

Hi @Ashwini Gaikwad
To end the debugging session, click at the upper-left of the cell or click at the top of the notebook. The debug session also automatically terminates after 30 minutes if the user does not use the debugging toolbar or debug console.
https://learn.microsoft.com/en-us/azure/databricks/notebooks/debugger

This behavior helps prevent long-running debugger sessions that could lead to unnecessary cluster resource usage and associated costs.

If you have observed instances where debugger sessions persist beyond the expected 30-minute timeout, it may indicate an issue with the auto-termination mechanism. In such cases, consider the following steps:

You can also set auto termination for a compute. During compute creation, you can specify an inactivity period in minutes after which you want the compute to terminate.

If the difference between the current time and the last command run on the compute is more than the inactivity period specified, Azure Databricks automatically terminates that compute.
https://learn.microsoft.com/en-us/azure/databricks/compute/clusters-manage

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.
Ashwini Gaikwad 130 Reputation points

2025-05-14T11:05:02.3633333+00:00

Hello @J N S S Kasyap thank you for your response.

On cluster there is already policy set which terminates the cluster after 45 mins of inactivity. But when %debug command is ran, the cluster doesn't stop at 45 mins or auto-terminates, you can see in the diagram below

First diagram shows the cluster is running for 3 days even though auto-termination for cluster is set to 45 mins

Second diagram states the cluster is running for more than 30 mins

Can you suggest what can be done here to auto-terminate the debug session after 30 mins?

Regards,

Ashwini
J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-05-14T14:40:14.56+00:00

@Ashwini Gaikwad
The %debug command initiates an interactive debugging session in Azure Databricks, which may be considered an active process by the cluster, preventing it from being marked as "inactive" for the purposes of auto-termination.
The auto-termination feature in Azure Databricks monitors activity such as Spark jobs but certain operations, including user-defined local processes or debugging sessions, may not be properly recognized as inactive, even if no explicit commands are running. This is particularly true in older Spark versions or specific runtime environments where activity reporting may have limitations.

Please consider the below options:

Older Spark or Databricks Runtime versions may have limitations in accurately reporting cluster activity, leading to premature or delayed termination during debugging sessions. Upgrade to the latest Databricks Runtime version (e.g., 12.2 LTS or higher, as of your context) to benefit from bug fixes and improvements in auto-termination logic.

The %debug session is not auto-terminating after 30 minutes, you can programmatically terminate it using the Databricks Clusters API to avoid manual intervention.Create a script or job to check for long-running clusters and terminate them after a set period.

If the %debug session is keeping the cluster alive, you can periodically run a lightweight command to reset the inactivity timer, ensuring the cluster recognizes activity and adheres to the auto-termination policy. Schedule a lightweight Spark job (e.g., spark.sql("SELECT 1").collect()) to run at regular intervals every 10–15 minutes during the debugging session

https://community.databricks.com/t5/data-engineering/how-to-prevent-my-cluster-to-shut-down-after-inactivity/td-p/29608

I hope this information helps. Please do let us know if you have any further queries.
J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-05-15T07:53:26.4133333+00:00

@Ashwini Gaikwad
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Ashwini Gaikwad 130 Reputation points

2025-05-15T15:08:03.89+00:00

@J N S S Kasyap all our clusters are running 14.3 or 15.4 LTS so we are using higher databricks runtime versions.

Can you let me know how can I stop the %debug feature after 30 mins approx so that it doesn't run for 3 days or so.

Can you share the command or a script or a notebook to auto-terminate the %debug activity or the cluster?

Regards,

Ashwini Gaikwad
Singireddi JagadishKumar 0 Reputation points Microsoft External Staff Moderator

2025-05-16T12:06:51.99+00:00

Hi Ashwini Gaikwad

Use Spark Watchdog Timeout

Databricks provides a Query Watchdog feature that can help terminate long-running commands, including %debug sessions, though it’s not officially documented for %debug. It’s worth testing in your environment.

Cluster-Level Configuration
Navigate to the Compute in your Databricks workspace.
Select your cluster and click Edit. Under Advanced Options > Spark Config, add:

spark.databricks.queryWatchdog.enabled= true

spark.databricks.queryWatchdog.timeout = 1800

This sets a 30-minute timeout (1800 seconds) for any command, including %debug.

Click Confirm and Restart the cluster.
Singireddi JagadishKumar 0 Reputation points Microsoft External Staff Moderator

2025-05-19T04:28:35.4966667+00:00

Hi @Ashwini Gaikwad

If you need a more customizable debug session with timeout control, we recommend using the approach shown in Command #2.

In this method, the %debug magic command is encapsulated within a function that accepts a user-defined timeout duration (in seconds). The function will automatically terminate the debug session once the specified timeout is reached, helping to avoid unintended cluster usage and associated costs.

Therefore, instead of directly using the %debug command, please use the code provided in Command #2. This ensures that the debug session will remain active only for the defined duration, after which it will be automatically terminated.

For eg, you may run the following two commands for better understanding on how it works,

#Command 1: def divide(a, b): return a / b result = divide(10, 0) # This raises ZeroDivisionError #Command 2: import time import threading import os import signal # Set the timeout duration (in seconds, e.g., 1800 seconds = 30 minutes) TIMEOUT_DURATION = 30 def terminate_debug_session(): """Terminate the debug session by sending a termination signal.""" print("Timeout reached. Terminating the debug session and running commands...") # Get the current process ID pid = os.getpid() # Send a SIGINT signal to the current process to interrupt the debug session os.kill(pid, signal.SIGINT) # Optionally, exit the notebook to prevent further execution dbutils.notebook.exit("Debug session terminated due to timeout.") # Start the timer in a separate thread timer_thread = threading.Timer(TIMEOUT_DURATION, terminate_debug_session) timer_thread.start() # Start debugging %debug # Cancel the timer if the debug session is completed manually timer_thread.cancel()
Singireddi JagadishKumar 0 Reputation points Microsoft External Staff Moderator

2025-05-20T04:33:34.1166667+00:00

Hi Ashwini Gaikwad
I wanted to follow up to check if you had an opportunity to review my response to your query. Please let us know if it addressed your concerns, and feel free to reach out with any additional questions or clarifications.

Share via

When using the %debug feature in databricks notebook, if the user forgets to cancel this command manually then it will continue to run until it is noticed or the cluster is manually terminated

Your answer