Azure Databricks Cloud Fetch Error

Question

IBM Cognos is unable to download large result sets from our Azure Databricks - Datalake platform. The reports runs fine for small result sets.

We are using 32bit simba odbc driver: version 2.06.26.1045 .

The issue seems to be with ADB Cloud Fetch Architecture, when we downgraded cluster runtime version to 7.3 LTS from 10.4 LTS, cognos is able to download large result sets.

Is there a way to disable cloud fetch on the cluster until we figure out what is causing failure with Cloud Fetch.

Error Message:

RQP-DEF-0177 An error occurred while performing operation 'sqlScrollBulkFetch' status='-232'.
UDA-SQL-0107 A general exception has occurred during the operation "fetch".
[Simba][Hardy] (35) Error from server: error code: '0' error message: '[Simba][Hardy] (134) File 6a2461e5-d225-4d08-9693-e3a9d0371348: A retriable error occurred while attempting to download a result file from the cloud store
but the retry limit had been exceeded. Error Detail: File 6a2461e5-d225-4d08-9693-e3a9d0371348: The result file URL had expired on 1666892082887 (Unix timestamp)'.
UDA-CUR-0000 Unable to fetch the row.

Answer 1

The issue is resolved.

Resolution: Open firewall between Cognos server and DBFS storage.

For large results sets , Databricks implemented cloud fetch feature to speed up downloads in BI tools such as Cognos, PowerBi etc. When the results set is larger than 1 MB Databricks stores multiple small files on DBFS storage and provides signed urls to the BI tools to download. In our case the traffic between cognos & DBFS storage was blocked.

Refer: https://learn.microsoft.com/en-us/azure/databricks/integrations/jdbc-odbc-bi#cloud-fetch-in-odbc