Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,818 questions
How do find the user/principal submitting my Spark Application?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 28.999999999999996%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJB%3C/text%3E%3C/svg%3E)
Jeff Born (J&CLT-ATL)
71
Reputation points
I've been trying to track down why I can't submit an application/run a Synapse Notebook, but some guid can. In the monitor section of Azure Synapse Workspace. I click on Apache Spark Applications and I see a column for submitter. I see my username which fails, but I also see a submitted described as a guid. How do I figure out the service principal/user associated with this guid? Also how specify which user should be used to submit an application when I run a notebook via a pipeline? Currently all pipelines submit applications using a default guid.
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth 1,045 Reputation points Microsoft Vendor2023-11-14T12:16:36.2433333+00:00 Thanks for reaching out to Microsoft Q&A.
- you can use the Azure CLI or Azure PowerShell. Here are the steps to do this using Azure CLI:
- Open Azure CLI and run the following command to get the details of the service principal associated with the GUID:
az ad sp show --id <GUID>
- Replace
<GUID>with the GUID you want to check. - This command will return the details of the service principal, including the display name, object ID, and other information.
To specify which user should be used to submit an application when you run a notebook via a pipeline, you can use the
spark.confconfiguration setting in the notebook. Here are the steps to do this:- Open the notebook you want to run in Azure Synapse Studio.
- Click on the "Settings" button in the toolbar and select "Advanced Settings".
- In the "spark.conf" section, add the following configuration setting:
spark.databricks.servicePrincipalId <service_principal_id>- Replace
<service_principal_id>with the object ID of the service principal you want to use to submit the application. - Save the notebook and run it via the pipeline. The application will be submitted using the specified service principal.
I hope this helps! please do Let us know if you have any further questions.
-
Jeff Born (J&CLT-ATL) 71 Reputation points
2023-11-14T18:13:59.64+00:00 I still can't track down who/what is submitting my Apache Spark Applications. It makes sense that if I run the notebook, that I'm the submitter. When I run the notebook via a pipeline the submitter is a GUID.
When I try to run the az command I'm getting the the following:
az ad sp show --id xxx Resource 'xxx' does not exist or one of its queried reference-property objects are not present.Is this GUID not a service principal? This submitter GUID that I get via a pipeline is the same whether I'm in my non prod, QA, or production environment.
-
phemanth 1,045 Reputation points Microsoft Vendor
2023-11-15T06:41:18.2666667+00:00 If the GUID you are seeing in the submitter column of the Apache Spark Applications is not associated with a service principal, it is possible that it is associated with a user account or a system account.
To find out more information about the GUID, you can try using the Azure AD Graph API or Microsoft Graph API to retrieve the details of the object associated with the GUID. Here are the steps to do this using the Azure AD Graph API:
Open the Azure AD Graph Explorer at https://graphexplorer.azurewebsites.net/.
Sign in with your Azure AD credentials.
Run the following query to retrieve the details of the object associated with the GUID:
https://graph.windows.net/<tenant_id>/directoryObjects/<object_id>?api-version=1.6Replace
<tenant_id>with the ID of your Azure AD tenant and<object_id>with the GUID you want to check.This query will return the details of the object associated with the GUID, including the object type, display name, and other information.
If the GUID is associated with a user account, you can try contacting the user to find out if they submitted the Apache Spark Application. If the GUID is associated with a system account, it may be more difficult to determine who or what submitted the application.
I hope this helps! please do Let us know if you have any further questions.
-
phemanth 1,045 Reputation points Microsoft Vendor
2023-11-16T14:49:29.4233333+00:00 @Jeff Born (J&CLT-ATL) We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Sign in to comment