Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,910 questions
Unable to fetch large data using DatabricksJDBC42-2.6.25.1044 jar
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 31%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGS%3C/text%3E%3C/svg%3E)
Ghosh, Sourav (893)
1
Reputation point
I'm trying to read data using DatabricksJDBC42-2.6.25.1044 jar. it's working perfectly fine with small datasets , however it's failing with the below error for larger datasets. I can observe the failure as soon as the table read completes at databricks cluster (DBR 9.1) side.
Note: Couldn't find relevant documentation related to this error and documents related to JDBC runtime parameters, like fetchSize, queryTimeout and so on. .
Exception in thread "main" java.sql.SQLException: [Databricks]DatabricksJDBCDriver The file <Blob storage URL> has not been downloaded successfully and the driver will not retry due to exceeding of the max retry limit 10, you can increase the max retry limit by setting MaxConsecutiveResultFileDownloadRetries.
at com.databricks.client.spark.jdbc.ResultFileDownloadManager.checkAndHandleDownloadError(Unknown Source)
at com.databricks.client.spark.jdbc.ResultFileDownloadManager.getNextDownloadedFile(Unknown Source)
at com.databricks.client.spark.jdbc.DowloadableFetchClient.fetchNRows(Unknown Source)
at com.databricks.client.hivecommon.api.HS2Client.fetchRows(Unknown Source)
at com.databricks.client.hivecommon.dataengine.BackgroundFetcher.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Caused by: com.databricks.client.support.exceptions.GeneralException: [Databricks]DatabricksJDBCDriver The file <Blob storage URL> has not been downloaded successfully and the driver will not retry due to exceeding of the max retry limit 10, you can increase the max retry limit by setting MaxConsecutiveResultFileDownloadRetries.
... 9 more
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT 26,011 Reputation points2022-05-26T21:53:40.097+00:00 Hello @Ghosh, Sourav (893) ,
Thanks for the question and using MS Q&A platform.As I understand, the ask is help with the cause of this error message, and request for any related documentation.
First I will try to dissect this error message.
The reader tried to read a blob, but failed. It tried again and again until it hit the set limit on retries.
Given that your process succeeded in smaller files, but not this large one, I speculate that the reader hit a time limit on how long a read can go before being deemed a failure. (This is to stop cases where an operation never finishes, and the computer hangs.)
There is also the possibility that something else caused the read to fail each time. To determine whether the cause is a timeout or something else, I would need more information. Logs and/or the duration of the operation would help.Could you please give a little more context? I am not an expert in JDBC, but this sounds like this might be Cloud Fetch?
As I understand, the Cloud Fetch uses the JDBC driver to make the database write the query results to a blob file. Then it gives a token to Databricks so Databricks can download the result. This last part sounds like the failure. What I would do, is look around inside the DBFS for the results file, and inspect it manually. Trying to open the results, outside the context of Cloud Fetch would reveal the cause of the error. Maybe permission? Maybe it wasn't written?
Sign in to comment