Azure Synapse Databricks notebook is not running

Question

I am building an integration:
Copy data1- Source: http, Sink: ADLS Gen 2
Notebook1: configured the workspace of azure databricks.

In databricks notebook having the code to pull and read the data from ADLS Gen2 raw directory and after performing some transformation loading the data into ADLS transformed directory. Databricks notebook all commands are running perfectly fine.

But when running the synapse pipeline its failing at notebook end with error:

{
    "errorCode": "3204",
    "message": "Databricks execution failed with error state: InternalError, error message: Cluster 0106-082338-jk5fnd4k was terminated while waiting on it to be ready: Cluster 0106-082338-jk5fnd4k is in unexpected state Terminated: INVALID_ARGUMENT(CLIENT_ERROR): databricks_error_message:Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 10, Current Usage: 8, Additional Required: 4, (Minimum) New Limit Required: 12. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22666e43b3-6184-40e8-9fdf-561360dc8ea2%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22cores%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:12,%22unit%22:%22Co ...\n***WARNING: message truncated. Skipped 282 bytes of output**\n\n. For more details please check the run page url: https://adb-4181940747004694.14.azuredatabricks.net/?o=4181940747004694#job/382562378777542/run/522321628328303.",
    "failureType": "UserError",
    "target": "Notebook1",
    "details": []
}

Answer 1

The key part of the message is "Operation could not be completed as it results in exceeding approved Total Regional Cores quota." This indicates that the operation you're trying to perform requires more computational resources than are currently allocated to you under your Azure subscription.

Try submitting a request for a quota increase. It provides a link (https://aka.ms/ProdportalCRP) where you can apply for an increase in your cores quota. You will need to increase the quota to at least 12 cores to meet your current requirements.

Based on this old thread :

each Azure subscription has is quotas for different Azure services.
Depending on the type of Azure subscription it is a hard hard quota limit, for instance for Free Trial subscriptions.
https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#managing-limits