Custom libraries (wheel) for ADF Databricks Python activity run on serverless compute
Krzysztof Przysowa
20
Reputation points
I want to be able to execute Python scripts (via Databricks Python) from Azure Data Factory using serverless compute.
Serverless compute does not support cluster level (compute scoped) libraries.
In databricks workflows, it is being done as "environments and libraries"
https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs#configure-environments-and-dependencies-for-non-notebook-tasks
I would like to be able to point to databricks workspace file location or volume file location for required library.
Please add this functionality.