Azure Databricks - java.lang.IllegalStateException: dbfs:/mnt/delta_checkpoints/offsets/5 exists before writing

Anonymous

I have built a streaming pipeline with spark autoloader.

Source Folder is a azure blob container.
We encountered a rare issue (could not replicate it). Below is the exception Message:

java.lang.IllegalStateException: dbfs:/mnt/delta_checkpoints/offsets/5 exists before writing

Please help on this with a resolution, as this looks like some known platform issue.

PRADEEPCHEEKATLA 91,866 Reputation points

2022-03-24T09:01:53.763+00:00

Hello @Anonymous ,

Welcome to the MS Q&A platform.

From the error message it looks like the checkpoint path already exists before writing. I would request you provide a new checkpoint path and see if that works.

In case, if you are experiencing the same error message again. Kindly share the complete command which you are trying?
PRADEEPCHEEKATLA 91,866 Reputation points

2022-03-28T06:56:04.683+00:00

Hello @Anonymous ,

Follow up to see if you have had a chance to see the previous response. We need the following information to understand/investigate this issue further.

Your answer