Synapse Pipeline - How do I specify identity for Pipeline Run? (SP/UAMI etc)

Question

I am working with Synpase Spark Pools in a controlled corporate environment. I have limited permission to query AAD but I can create UAMIs and assign them to Resources.

When I access my Synpase workspace I can create a Spark Job Definition to read some data from ADLS. Looking at the Apache Spark Applications list under the Monitor tab I can see that these jobs use my identity (tim.davies@Work .com) as the 'Submitter', and since I have given myself rx access to the data store these succeed.

Now if i create a Pipeline, and configure it to run my Spark Job Definition, it fails with an authorisation error. Going back to Apache Spark Applications list under Monitor I see that my Pipeline has a different Identity used as Submitter, which would explain why it is not authorised to access the data.

Firstly, I'm not sure which identity is now being used as Submitter, I don't recognise the UUID as either my Synapse Workspace SAMI or UAMI, (but I can't query AAD for more info).

However in general it occurs to me that I would probably like to be able to assign explicit UAMIs for my Pipelines to run under. Is this possible? Or is there a different model for managing this?

Answer 1

Hello @Davies, Tim-XT ,

Thanks for the question and using MS Q&A platform.

In Azure Synapse, when you create a pipeline, it is associated with a specific workspace. The identity that is used to submit a pipeline job is the identity of the workspace. This identity is known as the workspace's system-assigned managed identity (SAMI).

You can use the SAMI to grant access to resources in Azure that the workspace needs to access, such as Azure Data Lake Storage (ADLS). When you create a pipeline, the SAMI is used to submit the pipeline job and access the resources that are required for the pipeline to run.

It is not currently possible to specify a specific user-assigned managed identity (UAMI) to use when submitting a pipeline job. However, you can use role-based access control (RBAC) to manage access to resources in Azure Synapse. By assigning roles to the SAMI for the workspace, you can control which resources the workspace has access to and what actions it can perform on those resources.

If you need to access resources that are not in the same subscription as the workspace, you can use resource identity to grant access to those resources. Resource identity allows you to associate a resource with a SAMI or UAMI, which can then be used to access the resource.

If you need to use a specific identity to access resources when running a pipeline, you can use a combination of resource identity and RBAC to grant access to those resources. For example, you could create a UAMI and use resource identity to associate it with the resource that you need to access. You can then use RBAC to grant the UAMI the necessary permissions to access the resource.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is jhow you can be part of Q&A Volunteer Moderators