Long running Databricks notebook but cumulative processing time is short

James Hooi 0

Hi Microsoft experts,

I have a long running Databricks notebook tonight, and I can't see a reason why. My Data Factory pipeline which calls the notebook has been running reliably everyday for about 8 months, but this week it has had some problems and now today it is running for a long time (over an hour, still running - usually done in 10-15 minutes). What's strange is that the elapsed time according to the Databricks notebook cells is only a few minutes, so it doesn't accord with how much time has elapsed.

I'm concerned that I am getting charged for the Databricks cluster, yet it isn't running commands. Can anyone please advise?

ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee

2023-01-16T08:59:00.5033333+00:00
Hi James Hooi,

Thank you for posting query in Microsoft Q&A Platform.

What is your notebook activity status in ADF? Is it queued or in progress?

Which cluster is getting used in your linked service? Is same cluster is getting used by any other jobs as well at same time?

Did you tried cancel execution and trying to re-run? If yes, how it behaved?

Are you able to run this notebook directly in data bricks workspace without any issues?

Try increasing pool size and using different type of cluster such as on-demand cluster and see if that helps?

Please let me know how it goes.
James Hooi 0 Reputation points

2023-01-29T03:12:39.82+00:00
Hi @ShaikMaheer-MSFT

Sorry for the very slow reply. To respond to your questions:

It showed as "In Progress"

The cluster only runs one job at a time, it has not had memory issues before or since

(and 4.) Re-run and manual notebook run both work fine.

In short, the job ran fine before, has run fine since, yet on this one day it was stuck for a long time and presumably billing me!

Cheers,
James