Azure Databricks: "java.sql.SQLTransientConnectionException: elasticspark - Connection is not available, request timed out after 10000ms."

Question

Azure Databricks: "java.sql.SQLTransientConnectionException: elasticspark - Connection is not available, request timed out after 10000ms."

DisplayName42 56

We have a Data Factory pipeline which runs Azure Databricks notebooks. This pipeline has been working for months without issues. In the last 2 weeks we got the following error message already a few times: Run result unavailable: job failed with error message java.sql.SQLTransientConnectionException: elasticspark - Connection is not available, request timed out after 10000ms. Cause: elasticspark - Connection is not available, request timed out after 10000ms. The pipeline starts every 2 hours and runs 13 notebooks in parallel. My questions:

Is there a way to get more information about this error? Why are there no logs on the databricks run page?
What does "elasticspark" mean in this error message?

KranthiPakala-MSFT 46,822 Reputation points Microsoft Employee Moderator

2023-04-14T14:53:45.26+00:00
Hi @DisplayName42 ,

Welcome to Microsoft Q&A forum and thanks for reaching out here.

Is there a way to get more information about this error? Why are there no logs on the databricks run page? To get more information about this error, you can check the logs in the Azure Databricks workspace.
It is possible that the logs are not available on the Databricks run page because the error occurred before the logs were generated or because the logs were not captured due to a configuration issue. I would recommend to go to your cluster and see if you can find any information about the failure. For more information on where to find logs, please refer to these threads. Databricks Cluster Get Executor Logs After Completion
Databricks Audit Logs, Where the log files are stored? How to read them?

What does "elasticspark" mean in this error message?

It is likely the name of the Databricks cluster that you are using to execute the jobs. Could you please check and confirm if it is the same or a different cluster you used in the notebooks.

As you mentioned that 13 notebooks running in parallel could you please confirm if all the notebooks are using the same cluster? Since you have mentioned that there are 13 notebooks running parallel, it is possible that the issue is caused by the number of notebooks running in parallel. You may want to consider reducing the number of notebooks running in parallel to see if that resolves the issue. Or you can also try increasing the compute resources of the cluster that is being used to execute the parallel jobs. Running too many notebooks in parallel can cause resource contention and lead to issues such as timeouts and connection errors. This is because each notebook requires resources such as memory and CPU, and running too many notebooks at the same time can cause the resources to be spread too thin, leading to performance issues. Hope this helps. Let us know how it goes.

Thank you
DisplayName42 56 Reputation points

2023-04-17T08:53:03.5566667+00:00

Hi KranthiPakala-MSFT Thank you for the suggestions and the links. The Databricks notebook runs on a new cluster every time (created by the Data Factory activity) and I can not see those clusters in the Databricks compute tab. In the links you mentioned I can find information on how to configure logging to DBFS for new clusters that I create on the Databricks web ui. Is it possible to configure logging to DBFS for clusters which are created by a Data Factory pipeline activity? The name of the Databricks cluster is automatically generated by the Data Factory pipeline and different from "elasticspark" . We do not use that name anywhere. I could try to reduce the number of notebooks which run at the same time. However, as I mentioned: this worked for quite a long time without issues and I would like to find out more about the root cause for this error.