Databricks autoloader error java.lang.IllegalStateException

Anirudh Putrevu 6 Reputation points
2022-08-24T17:45:40.757+00:00

I am running 4 autoloaders on the same bucket with different checkpoint locations, stream names and using pathGlobFilter to select different files. I only get this error sometimes but other times the autoloader runs fine. Can someone help me understand what's going wrong here.

java.lang.IllegalStateException: The container in the file event {"backfill":{"bucket":"<name@container>","key":< a file in the container i'm not selecting via globPathFilter>","size":1512694,"eventTime":1661302803000,"newerThan$default$2":false}} is different from expected by the source: root@dbstorage<hex>.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,373 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Anirudh Putrevu 6 Reputation points
    2022-09-07T18:55:17.81+00:00

    I'm not facing this issue any more. I just restructured my program and ran the notebook independently rather than inside a different notebook.

    1 person found this answer helpful.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.