Getting error when running spark pool

Question

Getting error when running spark pool

Abhishek Ramamurthy 70

Every time i run the notebook i get this error:
InvalidHttpRequestToLivy: Your Spark job requested 12 vcores. However, the workspace has a 0 core limit. Try reducing the numbers of vcores requested or increasing your vcore quota. Quota can be increased using Azure Support request https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-concepts#workspace-level HTTP status code: 400. Trace ID: c1c645bd-fa19-4eeb-b8ae-08a885f5c693.

I've tried with different core sizes like large 16vCores but still its the same. Can anyone please help me with it? thanks User's image

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2024-02-26T08:34:10.62+00:00

@Abhishek Ramamurthy - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Abhishek Ramamurthy 70 Reputation points

2024-02-26T16:42:46.43+00:00

Hey,
I tried contacting sales team, but the chat bot is not taking me to any representative. I think it has some bugs kindly look into it. I also tried calling the customer service number, but there's no answer as well. So, in short no, my problem was not solved, and I did not receive any proper assistance.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2024-02-28T05:27:55.87+00:00

@Abhishek Ramamurthy - Apologize for the incovience. Could you please try to open a support ticket to resolve the problem: Create an Azure support request.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2024-03-11T04:08:24.4433333+00:00

@Abhishek Ramamurthy - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Abhishek Ramamurthy 70 Reputation points

2024-03-12T00:14:50.66+00:00

Hey, since student accounts are not eligible to core upgrade, I continued my learning without practically implementing spark notebook. If you have any other solution please let me know. Thanks!
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2024-03-12T04:24:48.3366667+00:00

@Abhishek Ramamurthy - There is no other workaround to resolve this issue, unless you are using the miminum required cores which are available under your subscription.

Hope this helps.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2024-03-14T09:46:23.8133333+00:00

@Abhishek Ramamurthy - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2024-03-18T04:49:17.8766667+00:00

@Abhishek Ramamurthy - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Accepted answer

0 additional answers

Your answer

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2024-02-26T08:34:10.62+00:00

@Abhishek Ramamurthy - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Abhishek Ramamurthy 70 Reputation points

2024-02-26T16:42:46.43+00:00

Hey,
I tried contacting sales team, but the chat bot is not taking me to any representative. I think it has some bugs kindly look into it. I also tried calling the customer service number, but there's no answer as well. So, in short no, my problem was not solved, and I did not receive any proper assistance.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2024-02-28T05:27:55.87+00:00

@Abhishek Ramamurthy - Apologize for the incovience. Could you please try to open a support ticket to resolve the problem: Create an Azure support request.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2024-03-11T04:08:24.4433333+00:00

@Abhishek Ramamurthy - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Abhishek Ramamurthy 70 Reputation points

2024-03-12T00:14:50.66+00:00

Hey, since student accounts are not eligible to core upgrade, I continued my learning without practically implementing spark notebook. If you have any other solution please let me know. Thanks!
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2024-03-12T04:24:48.3366667+00:00

@Abhishek Ramamurthy - There is no other workaround to resolve this issue, unless you are using the miminum required cores which are available under your subscription.

Hope this helps.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2024-03-14T09:46:23.8133333+00:00

@Abhishek Ramamurthy - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2024-03-18T04:49:17.8766667+00:00

@Abhishek Ramamurthy - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

@Abhishek Ramamurthy - Thanks for the question and using MS Q&A platform.

Every Azure Synapse workspace comes with a default quota of vCores that can be used for Spark. The quota is split between the user quota and the dataflow quota so that neither usage pattern uses up all the vCores in the workspace. The quota is different depending on the type of your subscription but is symmetrical between user and dataflow. However if you request more vCores than are remaining in the workspace, then you will get the following error: InvalidHttpRequestToLivy: Your Spark job requested 12 vcores. However, the workspace has a 0 core limit. Try reducing the numbers of vcores requested or increasing your vcore quota. HTTP status code: 400..

As per the error message, the number of V-cores are exhausted on the spark session. When you define a Spark pool, you are effectively defining a quota per user for that pool.

The vcores limit depends on the node size and the number of nodes. To solve this problem, you have to reduce your usage of the pool resources before submitting a new resource request by running a notebook or a job.

(or)

Please scale down the node size and the number of nodes. User's image

Note: Azure free trial/Student/Pass subscription are not eligible for a quota request. You need to upgrade to a Pay-As-You-Go subscription to increase the quota and credit card is the necessary to upgrade to pay-as-you-go subcription.

If you still have questions, you can chat with sales team to get more information: https://azure.microsoft.com/en-in/free/students

User's image

To resolve this issue, you need to request a capacity increase via the Azure portal by creating a new support ticket.

Step1: Create a new support ticket and select issue type as Service and subscription limits (quotas) and quota type as Azure Synapse Analytics.

Step2: In the Details tab, click on Enter details and choose quota type as Apache Spark (vCore) per workspace , select workspace, and request quota as shown below.

Step3: Select support method and create the ticket.

For more details, refer to Apache Spark in Azure Synapse Analytics Core Concepts.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

Getting error when running spark pool

0 additional answers

Your answer