How is the divison in terms of Spark applications submitted from a pipeline handled?

asciibscii 45

In a Synapse pipeline I have 4 different notebooks that are attached to the same Spark pool with 1 executor each. The spark pool has 3-8 nodes so it should have enough capacity to run all notebooks on the same Spark Instance. The pipeline has a trigger to run once every hour. I thought because there is enough capacity in the spark pool and that they are being started by the same pipeline the submitter for the Apache Spark Application would be the same for all of them, but what I am seeing is that the pipeline has 2 different submitters and that 3 notebooks get och submitter while 4th notebook gets the other pipeline submitter. How can I make it so that all Notebooks get the same submitter? Or rather what is the issue here/how does it work in terms of user/submitter to a Spark pool from a synapse pipeline?

BhargavaGunnam-MSFT 26,136 Reputation points Microsoft Employee

2023-01-19T00:19:09.0766667+00:00

Hello @asciibscii,

Welcome to the MS Q&A platform.

When you submit a notebook job (J1) that uses 10 nodes, a spark instance(S1) is created. You now submit another notebook job (J2) that uses 10 nodes. If there is still capacity in the pool, J2 will be processed by the spark instance S1.

If J2 asked for 11 nodes, there would not have been capacity in Spark pool or on the spark instance S1; in this case, if J2 comes from a notebook, then the job will be rejected; if J2 comes from a batch job, then it will be queued.

If you are trying to submit the notebook via the pipeline, then workspace system-assigned managed identity will be used. (submitter)

If you are manually submitting the notebook, then your AAD identity will be used to submit the notebook.

If you are trying to submit all 4 notebooks via the pipeline, they will be submitted by the same account, i.e, workspace system-managed identity.

If you are seeing a different behavior, please send us more details with the screenshot.

Please see this reference document: [https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/understand-the-identity-used-during-component-execution-in/ba-p/2567863

I hope this helps. Please let us know if you have any further questions.
asciibscii 45 Reputation points

2023-01-19T13:45:29.8766667+00:00

Hello @BhargavaGunnam-MSFT

I feel like I am seeing a different behavior, This is a screenshot from one of triggerruns from a couple of days ago where 4 different notebooks were submitted from a pipeline where 3/4 notebooks got the same submitter for the different apache spark applications.
Here the spark pool that it attached to is a small pool (4 vCores / 32 GB) with 3-8 nodes. From my understanding the pool has up 7 executors that can be distributed across different Spark jobs. At the time of this pipeline run I had that Notebook 1, NB1, got 1-2 Executors with dynamic allocation of executor enabled, NB2 and NB3 both had dynamic allocate executors set to Disabled and got 1 executor each. NB4 had that it got 2 executors with dynamic allocation disabled.

From my understanding there should be enough executors in the pool for all of the notebooks to share, since we're not exceeding 7 executors when running the pipeline. Unless it's so that we have to account for the driver executor. But even then I tried to run a pipeline run where 3 notebooks were run inside of a pipeline where they got 1 executor each, so counting with the driver executor that would have been 6 executors out of 7 and I still saw that I was getting different submitters for the apache spark application under the monitor tab.
On the topic of it begin the workspace managed identity which is the user that submits the spark job, does this mean that what I see as a submitter to an apache spark application should match the Managed Identity object ID for the synapse workspace? Because it does not match either of the 2 submitter ID's that I can see under the monitoring tab for apache spark applications from pipeline runs.
BhargavaGunnam-MSFT 26,136 Reputation points Microsoft Employee

2023-01-23T23:41:50.3666667+00:00

Hello @asciibscii,

with small pool (4 cores/32GB) and 8 nodes , The max Vcores = 4*8 = 32 Vcores available

for NB1 = 2 executors + 1 driver node = 2*4 + 4 = 12 Vcores (max vcores available for NB1)

consider NB1 uses the max vcores that were assigned

Remaining vcores = 32 - 12 = 20 Vcores

for NB2, Nb3 each has 1 executors

NB2 = 4+ 4(driver node) = 8 vcores

NB3 = 4+ 4(driver node) = 8 vcores

Total Vcores used so far = 12+8+8 = 28

Remaning Vcores = 32- 28 = 4 Vcores.

The same spark instance will be used here for the three notebooks.

for the 4th notebook, as there are only 4 vcores available, for the fourth notebook if this request comes as a notebook, it will be rejected. Or if this comes as a batch job then it will be queued.

For the 4th notebook to run, you will need to have 8 cores ( as 4 cores are required for the driver node)

For your other question, As per the documentation, my understanding is if a user submits the notebook, then AAD of the user will be used.

I never tested this from my end for this scenario, but I can test this and let you know.

[https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/understand-the-identity-used-during-component-execution-in/ba-p/2567863

Please let me know if you have any further questions
asciibscii 45 Reputation points

2023-01-24T12:31:43.9833333+00:00

Hi @BhargavaGunnam-MSFT thanks for the very good answer!

I now understand what should happen. I still think I was seeing some wrong behavior from the runs then. The trigger which is set up with my pipeline runs twice and hour and from those two runs it would run 3 notebooks in one and 4 notebooks in the other.

So here it was submitting NB1, NB4 and NB2 in this particular order. So with a small pool 4vCores/32GB with 3-8 nodes. The max vCores = 4*8= 32 vCores available. Driver being the same size as executor for all notebooks.

For NB1 (1-2 executors) with max vcores available for NB1 = 2 executors + 1 driver = 2x4+4=12 vCores; using 3 nodes out of max 8 nodes

vCores left in the pool: 32-12=20 vCores

For NB4 (2 executors), NB4 = 2 executors + 1 driver = 2x4+4=12vCores

vCores left in the pool: 20-12=8vcores

For NB2 (1 executor), NB2 = 1 executor + 1 driver = 1x4+4=8vCores

vCores left in the pool: 8-8=0

So what I thinking is that there should be enough resources to run these 3 notebooks in the same Spark Instance, with the same submitter for the Apache Spark Application. But as you can see in the photo they don't all have the same submitter.

I have read the documentation that you linked and I also agree with you that if a user submits a notebook then the AAD of that user will be used for authentication and it seems like the submitter for creating a Spark Instance. And I can see that Workspace managed identity is supposed to be used when executing notebooks in a pipeline, which is this case. But I am dubious that that identity is what is the submitter for the apache spark applications which you can see under the monitor tab.

The actual Workspace managed identity for my synapse resource has a completely different ID than the two from above,
BhargavaGunnam-MSFT 26,136 Reputation points Microsoft Employee

2023-01-25T17:20:06.67+00:00

Hello @asciibscii,

Thank you for the details. Sorry, I didn't get a chance to test this scenario from my end. I will test this and get back to you with my findings.

Appreciate your patience.
BhargavaGunnam-MSFT 26,136 Reputation points Microsoft Employee

2023-01-26T17:57:43.8+00:00

Hello @asciibscii,

I was able to reproduce this from my end. When I run the notebook manually, I see my user ID as the submitter. But when the notebook runs via the trigger run or debug run, it uses some ID(which I don't find in the AAD). This is not using the synapse workspace Managed Identity. But my understanding from the documentation is it is supposed to use the MSI of the workspace.

I reached out to my internal team for clarification on this. I will get back to you as soon as I hear from them.

Appricaite your patience.

Accepted answer

BhargavaGunnam-MSFT 26,136 Reputation points Microsoft Employee

2023-01-27T23:05:38.52+00:00

Hello @asciibscii,

<update>

Per my discussion with the internal team, they confirmed that this is not a workspace MSI or user objectID.

This is a system principal ID at the service level.

The system principal ID is a unique identifier for the Azure service instance that is enabled with a system-assigned managed identity. It is used to authenticate the service instance with Azure resources that support Azure AD authentication.

The system principal ID is automatically created by Azure when you enable a system-assigned managed identity on the service instance.

Please refer to the below document on how to view the system principle.

https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/how-to-view-managed-identity-service-principal-portal#view-the-service-principal

Below is a screenshot of the notebook job submitter. I can find this ID in the AAD on the enterprise applications.

Unfortunately I don't have these details on the documentation page. But I will request PG to document these details.

I hope this helps. Please let me know if you have any further questions.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions
Please sign in to rate this answer.
BhargavaGunnam-MSFT 26,136 Reputation points Microsoft Employee

2023-02-01T00:38:53.2966667+00:00

Hello @asciibscii,

I am checking to see if you got a chance to look into my above answer.

asciibscii 45 Reputation points

2023-02-15T12:42:13.7566667+00:00

Hello @BhargavaGunnam-MSFT

Yes I have looked at the answer now, thank you for finding the answer as to what it is, much appreciated.

BhargavaGunnam-MSFT 26,136 Reputation points Microsoft Employee

2023-02-15T17:12:06.11+00:00

Thank you, @asciibscii
Sign in to comment

How is the divison in terms of Spark applications submitted from a pipeline handled?

0 additional answers