DLT pipeline fails with an error "Error while reading file abfss:REDACTED_LOCAL_PART@stanalyticsdeastus2001.dfs.core.windows.net/" on Databricks"

Apoorva Nutula 1

I have delta live tables pipeline on Databricks that creates tables for different sources using one notebook. This pipeline is failing for one particular source with the error mentioned in the title. It is working just fine for other source.

PRADEEPCHEEKATLA-MSFT 77,751 Reputation points Microsoft Employee

2022-12-07T06:23:01.25+00:00

Hello @Apoorva Nutula ,

Thanks for the question and using MS Q&A platform.

Could you please share the stack trace of the error message which you are experiencing?
Apoorva Nutula 1 Reputation point

2022-12-07T18:44:13.093+00:00

org.apache.spark.sql.streaming.StreamingQueryException: Query raw_TriMED_CHARGE_HISTORY [id = fdf08817-2287-4f96-a453-384e3d06b926, runId = 820dffab-3979-40b3-a84f-029e586e6ce9] terminated with exception: Job aborted.
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExec
com.databricks.sql.io.FileReadException: Error while reading file abfss:REDACTED_LOCAL_PART@stanalyticsdeastus2001.dfs.core.windows.net/LandingZone/RCM/TriMED/Florida/CHARGE_HISTORY/2022/12/07/data_574d2570-8103-4fda-bd33-2e3de978f034_8169dfe1-3b76-4abe-a2ee-792d21cea74c.parquet.
PRADEEPCHEEKATLA-MSFT 77,751 Reputation points Microsoft Employee

2022-12-12T06:32:20.047+00:00

Hello @Apoorva Nutula ,

We need complete stack trace which contains caused by in the error message?
PRADEEPCHEEKATLA-MSFT 77,751 Reputation points Microsoft Employee

2022-12-16T09:12:48.447+00:00

Hello @Apoorva Nutula ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond with more details and we will try to help.