Share via

DBT databricks job is stuck for a long time

Gil Shinar 0 Reputation points
2023-10-29T14:43:48.1033333+00:00

Hi

I've created a DBT databricks job and when I run, it gets stuck for a long time. The last message I can see in the output is:
**Opening a new connection, currently in state init
**
I have tried a few thing I read about with no success. My best bet is that the SQL warehouse fails to start. After a few minutes I see the following message:
**Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you.
**
The above message doesn't help me as it has no real issue in it.

Any clue on this issue?

Thanks

Gil

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,080 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Gil Shinar 0 Reputation points
    2023-11-01T07:33:48.28+00:00

    I recreated the databricks (there's no other way to solve that). If it was a production databricks workspace it was a disaster!
    I have created a VM with static public IP and added this IP to the IP access list. Hopefully it'll become the last resort in case all other IPs will be replaced.
    The above is an ugly workarround. No matter where I have looked, the only solution is to have private endpoint which requires other resources (peered VET + VPN) that I do not want to get into in this project.

    As for the original issue, the issue, as mentioned above, was the SQL warehouse that couldn't start. I have created a new serverless SQL warehouse and added it to the job instead of the starter warehouse that was available and problem solved

  2. PRADEEPCHEEKATLA-MSFT 85,746 Reputation points Microsoft Employee
    2023-11-03T05:37:10.2+00:00

    @Gil Shinar - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

    Ask: DBT databricks job is stuck for a long time.

    Solution: I recreated the databricks (there's no other way to solve that). If it was a production databricks workspace it was a disaster!
    I have created a VM with static public IP and added this IP to the IP access list. Hopefully it'll become the last resort in case all other IPs will be replaced.
    The above is an ugly workarround. No matter where I have looked, the only solution is to have private endpoint which requires other resources (peered VET + VPN) that I do not want to get into in this project.

    As for the original issue, the issue, as mentioned above, was the SQL warehouse that couldn't start. I have created a new serverless SQL warehouse and added it to the job instead of the starter warehouse that was available and problem solved.

    If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

    If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

    Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

    0 comments