Not able to execute databricks delta live table "PERMISSION_DENIED"

Felipe Regis E Silva 46

Hi,

My team is having issues to execute delta live tables (dlt) pipelines using our workspace. From requirements documentation, I noticed delta live table users should have cluster creation permission and I created a cluster policy they can use at this moment. After further analysis, I found 02 points I'd like share and ask support for:

The first is about a cluster policy that is not working for users when building dlt pipelines and with pretty much all settings they need at this point is attached. They tested it and they are able to create and run notebooks using a cluster through databricks workspace UI directly. However, when they try to run a dlt pipeline directly on the delta live tables panel, they get this error:

On the other hand, I'm workspace admin and I'm able to run the dlt pipeline without issues. We use a databricks premium account and users I mentioned belong to group with "CAN USE" permission to the cluster policy I created. I also tried to simplify configs for testing purposes and created another cluster policy to limit min and max number of drivers and workers only and that didn't work as well.

The second point is about a dlt pipeline settings that it's not working. There were a few parameters I provided that were ignored when I ran a dlt pipeline. After a pipeline run and checking dlt execution job, I noticed databricks runtime version used was below the one I provided in the edit pipeline settings, under configurations. I defined "pipelines.dbrVersion = 10.5". So I presume, it's not working. I'm not able to define a specific cluster type for development purposes.

Please, does anybody can help?

Cluster Policy created: 210926-dlt-cluster-policies.txt

Best regards,

Felipe Regis.

PRADEEPCHEEKATLA-MSFT 76,511 Reputation points Microsoft Employee

2022-06-14T08:32:28.367+00:00

Hello @Felipe Regis E Silva ,

Thanks for the question and using MS Q&A platform.

As mentioned in the cluster creation permission, make sure user must be an Admins or Can manage permission to create the clusters.

Cluster access control within the Admin Console allows admins and delegated users to give fine-grained cluster access to other users. There are two types of cluster access control:

Cluster creation permission: Admins can choose which users are allowed to create clusters.

A user who has the Can manage permission for a cluster can configure whether other users can attach to, restart, resize, and manage that cluster from the cluster list or the cluster details page.

Hope this will help. Please let us know if any further queries.
Felipe Regis E Silva 46 Reputation points

2022-06-14T16:37:14.187+00:00

Thanks @PRADEEPCHEEKATLA-MSFT . They are not admin users.

About granting them manage permissions, I don't think it's sth in a specific cluster or in regards to cluster configuration settings because databricks delta live table feature provisions job clusters automatically on each pipeline run as defined in the dlt configuration settings.

From documentation, I noticed dlt pipelines are run with the same principal and permissions of the dlt owner or the one who created it.

As the users I mentioned have "Can Use" permission through a cluster policie I created and they are still getting the error message, I believe the issue persists and they can not run dlt pipelines on our azure databricks workspace.

Regards,

FRS
PRADEEPCHEEKATLA-MSFT 76,511 Reputation points Microsoft Employee

2022-06-16T05:22:38.827+00:00

Hello @Felipe Regis E Silva ,

I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us.
PRADEEPCHEEKATLA-MSFT 76,511 Reputation points Microsoft Employee

2022-06-20T07:04:53.013+00:00

Hello @Felipe Regis E Silva ,

Did you get a chance to open a support ticket on this issue?
Felipe Regis E Silva 46 Reputation points

2022-06-20T20:26:26.477+00:00

Hi @PRADEEPCHEEKATLA-MSFT

Did you mean open a support ticket on the Azure Portal about it? Not yet. Is it the best way to get help, right?

Thanks,

Felipe Regis.
PRADEEPCHEEKATLA-MSFT 76,511 Reputation points Microsoft Employee

2022-06-22T08:37:36.197+00:00

Hello @Felipe Regis E Silva ,

Yes, you are correct. You need to open a support ticket from Azure Portal.
Amrutha K 0 Reputation points

2023-03-27T08:49:39.3133333+00:00

@Felipe Regis E Silva , is this issue resolved? What is the fix for this?