deadlock in databricks

Question

deadlock in databricks

Shambhu Rai 1,411

Hi Expert,

I am running '1st' notebook for every one minute using Devops schedular if it delayed in running due to lots of data then on 2nd minute schedule should wait to finish 1st notebook ... can we add some deadlock or threadlock

i.e. notebook load should not repeat or wait to finish the notebook and then it should run again

Shambhu Rai 1,411

i got below code but not sure how it works

while len(spark.streams.active) > 0:
  spark.streams.resetTerminated() # Otherwise awaitAnyTermination() will return immediately after first stream has terminated
  spark.streams.awaitAnyTermination()

1 answer

Your answer

Shambhu Rai 1,411 Reputation points

2023-12-29T06:11:01.94+00:00

i got below code but not sure how it works

while len(spark.streams.active) > 0: spark.streams.resetTerminated() # Otherwise awaitAnyTermination() will return immediately after first stream has terminated spark.streams.awaitAnyTermination()

Answer 1

ShaikMaheer-MSFT 38,631 Microsoft Employee Moderator

Hi

Thank you for posting query in Microsoft Q&A Platform.

It seems like you have a question about scheduling notebooks in Azure DevOps and how to avoid running the same notebook multiple times if it takes longer than expected to complete.

One way to achieve this is by using a locking mechanism to prevent multiple instances of the same notebook from running simultaneously. You can use a file or a database table as a lock to ensure that only one instance of the notebook is running at a time.

Here's an example of how you can implement this:

At the beginning of the notebook, check if the lock file or database table exists. If it does, exit the notebook. If it doesn't, create the lock file or insert a record into the database table.

At the end of the notebook, delete the lock file or delete the record from the database table.

This way, if the notebook is already running, the second instance will exit immediately without doing anything. Once the first instance completes, the lock will be released and the next scheduled instance of the notebook will be able to run.

I hope this helps! Let me know if you have any further questions.

Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.

Shambhu Rai 1,411 Reputation points

2024-01-05T19:04:21.2233333+00:00

Sir, either you answer the question with technical thread or di not answer.. It us request
ShaikMaheer-MSFT 38,631 Reputation points Microsoft Employee Moderator

2024-01-08T04:47:36.7566667+00:00

Hi Shambhu Rai,

Any hard dependency with devops here?? If not, you can consider using ADF also here. In ADF you can try to run notebooks one after another in pipeline. Also, you can consider triggering run this ADF pipeline from DevOps tasks also might be by using PowerShell scripts.

Hope this helps.

Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.

Share via

deadlock in databricks

1 answer

Your answer