My team is having issues to execute delta live tables (dlt) pipelines using our workspace. From requirements documentation, I noticed delta live table users should have cluster creation permission and I created a cluster policy they can use at this moment. After further analysis, I found 02 points I'd like share and ask support for:
- The first is about a cluster policy that is not working for users when building dlt pipelines and with pretty much all settings they need at this point is attached. They tested it and they are able to create and run notebooks using a cluster through databricks workspace UI directly. However, when they try to run a dlt pipeline directly on the delta live tables panel, they get this error:
On the other hand, I'm workspace admin and I'm able to run the dlt pipeline without issues. We use a databricks premium account and users I mentioned belong to group with "CAN USE" permission to the cluster policy I created. I also tried to simplify configs for testing purposes and created another cluster policy to limit min and max number of drivers and workers only and that didn't work as well.
- The second point is about a dlt pipeline settings that it's not working. There were a few parameters I provided that were ignored when I ran a dlt pipeline. After a pipeline run and checking dlt execution job, I noticed databricks runtime version used was below the one I provided in the edit pipeline settings, under configurations. I defined "pipelines.dbrVersion = 10.5". So I presume, it's not working. I'm not able to define a specific cluster type for development purposes.
Please, does anybody can help?
Cluster Policy created: 210926-dlt-cluster-policies.txt