Databricks throws [INTERNAL_ERROR] during the execution of Copy command

Question

Environment Details:

JDBC Used : Databricks JDBC v2.6.29

Cluster: Databricks Runtime Version 12.2 LTS

Cluster ID: adb-3421891622900168.8

Workspace Region : EastUS

Time Frame of the Error: August 20 to August 24

Issue:

Databricks throws the following exception often during the execution of copy command and for the same table.

But on retry the same query succeeds.

[Databricks]DatabricksJDBCDriver ERROR processing query/statement. Error Code: 0, SQL state: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.sql.AnalysisException: [RequestId=1459328e-ab5c-4d95-aa94-7a23b2eb56c0 ErrorClass=INTERNAL_ERROR] at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:48)

COPY INTO `prod_datalake`.`banner`.`rs_STAGEONE` FROM (SELECT CAST(_c0 AS STRING) AS `rs_aidy_code`,
CAST(_c1 AS BIGINT) AS `rs_pidm`,
..
..
..
130 columns
..
..
CAST(_c211 AS BOOLEAN) AS `rs_cfh_ind_2`
FROM 'abfss://******@adlspazdatalake.dfs.core.windows.net/.striim/DEA/trg_azure_databricks_cdc_parq11/prod_datalake.banner.rorstat/prod_datalake.banner.rorstat498.csv'
WITH ( CREDENTIAL (AZURE_SAS_TOKEN = [redacted] ))) FILEFORMAT = CSV FORMAT_OPTIONS('multiLine'='true','escape'='"') COPY_OPTIONS('force' = 'true')

There is no clear information about the error except for the error class INTERNAL_ERROR

The documentation is also not clear on this error class. Can you please provide more info on this error like the possible scenarios in which this exception would occur?

Attaching the full stack trace from the application and the JDBC driver.

Errors.txt