I'm not facing this issue any more. I just restructured my program and ran the notebook independently rather than inside a different notebook.
Databricks autoloader error java.lang.IllegalStateException
Anirudh Putrevu
6
Reputation points
I am running 4 autoloaders on the same bucket with different checkpoint locations, stream names and using pathGlobFilter to select different files. I only get this error sometimes but other times the autoloader runs fine. Can someone help me understand what's going wrong here.
java.lang.IllegalStateException: The container in the file event {"backfill":{"bucket":"<name@container>","key":< a file in the container i'm not selecting via globPathFilter>","size":1512694,"eventTime":1661302803000,"newerThan$default$2":false}}
is different from expected by the source: root@dbstorage<hex>
.