notebook-scoped lib failing to find the dependent cluster-scoped lib

Question

Our app hit an issue with ModuleNotFoundError recently with Databricks Runtime 6.4 and it looks this is a databricks issue, so wanted to ask about that.

The problem can happen as follows:

1. Two libraries A.B and A.C; A.C depends on A.B
2. Install A.B as cluster-scoped lib and it goes to default location, e.g. /databricks/python/lib/python3.7/site-packages/A/B
3. Install A.C as notebook-scoped lib via dbutils.library.installPyPI, this goes to virtual env location, e.g., /local_disk0/pythonVirtualEnvDirs/virtualEnv-ead9164f-61e5-4c26-9374-374bfb80263c/lib/python3.7/site-packages/A/C. And A.B was not installed because A.B with same version was already installed in cluster scope.
4. When we import A.C in notebook, A.B needs to be imported as well but this will fail with ModuleNotFoundError because A.B is not found in the virtual environment.

We hit this issue with A.B = "azure-core==1.8.2" and A.C = "azure-kusto-data==0.0.24". Our current workaround is to either install both packages together in cluster or both in notebook session.

Could you let us know if this is indeed a databricks issue and the plan to fix it? Thank you.

Answer 1

Hello @Tri Tran ,

We have discussed the same aks here : https://github.com/MicrosoftDocs/azure-docs/issues/64000

Thanks
Himanshu