Lakeflow Connect CDC Gateway stuck in WAITING_FOR_RESOURCES due to EDv4 QuotaExceeded (Classic Compute)

Question

Lakeflow Connect CDC Gateway stuck in WAITING_FOR_RESOURCES due to EDv4 QuotaExceeded (Classic Compute)

Naga Vardhan Sankar Raparthi 0

Hi,

I'm trying to create a Lakeflow Connect CDC ingestion pipeline from Azure SQL Database to Databricks.

The pipeline fails before source validation because the CDC Gateway compute never starts. The managed cluster remains in PENDING/WAITING_FOR_RESOURCES and the event log shows:

QuotaExceeded: Operation could not be completed as it results in exceeding approved standardEDv4Family Cores quota.

Current Limit: 0

Additional Required: 4

What I've verified:

Azure SQL Database is online and accessible.
The CDC Gateway uses pipeline-managed Classic Compute.
Driver and worker node types are locked, so I cannot change the VM family.
EDv4 is deprecated in Azure, and the quota request option is disabled/greyed out.
I checked Azure quotas and confirmed EDv4 quota is 0.
I tried creating new Databricks workspaces in multiple regions (Central India and UAE), but the gateway still attempts to provision EDv4 and fails with the same error.
Other VM families have available quota, but the managed gateway always requests EDv4.

Has anyone encountered this recently? Is there a way to configure the Lakeflow Connect CDC Gateway to use a supported VM family (such as EDv5/EDSv5), or is this a known issue with the current managed Classic Compute provisioning?

Thanks!

Peram Venkateswara Reddy 5 Reputation points Microsoft External Staff Moderator

2026-08-07T04:35:21.2166667+00:00
Hi @Naga Vardhan Sankar Raparthi ,

Hi Naga,

Good diagnosis — the failure is entirely on the gateway's classic compute, not Azure SQL or CDC configuration. But the node types aren't actually locked: they're fixed in the UI, and settable through a compute policy or the Pipelines API.

Why it requests EDv4

The documented gateway compute policy pins:

{ "driver_node_type_id": { "type": "fixed", "value": "Standard_E64d_v4" }, "node_type_id": { "type": "fixed", "value": "Standard_F4s" } }

Standard_E64d_v4 is EDv4. Since your quota there is 0, provisioning fails before anything else runs.

Fix: create a custom compute policy with a different driver family

Per the SQL Server connector requirements, the gateway accepts "unrestricted permissions to create clusters, or a custom policy (API only)." Create a Job Compute policy with the required overrides, plus your own node types:

{ "cluster_type": { "type": "fixed", "value": "dlt" }, "num_workers": { "type": "unlimited", "defaultValue": 1, "isOptional": true }, "runtime_engine": { "type": "fixed", "value": "STANDARD", "hidden": true }, "driver_node_type_id": { "type": "fixed", "value": "Standard_E8ds_v5" }, "node_type_id": { "type": "fixed", "value": "Standard_F4s_v2" } }

Pick driver/worker SKUs from families where you already have quota. Two constraints from the docs:

Minimum 8 cores total for efficient extraction from the source.

Databricks recommends the smallest possible worker nodes — worker size doesn't affect gateway performance, so keep the workers small and put the capacity in the driver.

Then attach the policy to the gateway pipeline via the Pipelines API:

{ "clusters": [ { "label": "default", "policy_id": "<your-policy-id>", "apply_policy_default_values": true } ] }

"apply_policy_default_values": true is required for the policy defaults to actually take effect.

In parallel — the quota route is still viable

Note your error says Additional Required: 4, not 64. That's a very small ask. If the portal's quota-increase option is greyed out for EDv4 in your region, raise an Azure support request under Service and subscription limits (quotas) → Compute-VM (cores-vCPUs) rather than using the self-service form. Request +4 (or +8) standardEDv4Family cores in your target region and reference this error text. Greyed-out self-service usually means the family needs manual approval in that region, not that it's unobtainable.

Before choosing a region, verify capacity

az vm list-skus --location centralindia --size Standard_E --all --output table az vm list-skus --location centralindia --size Standard_F --all --output table

Look for rows where Restrictions = None. This avoids the pattern of moving workspaces between regions and hitting the same wall — quota being available isn't the same as the SKU being unrestricted.

Recommended order: custom compute policy first (it unblocks you today without waiting on Azure), quota ticket in parallel as a fallback.

References

Ingest data from SQL Server — requirements and gateway compute policy

Configure classic compute for pipelines

Managed database connectors overview

One caution: don't manually stop a running gateway — it must run continuously so change logs aren't truncated at the source. Re-deploying with the new policy is fine at this stage since it never started.

Kind Regards,
Microsoft Support Team.
Naga Vardhan Sankar Raparthi 0 Reputation points

2026-08-07T10:39:04.1233333+00:00

Thank you for the detailed explanation and guidance.

I was able to resolve the issue. I updated the ingestion gateway using the Databricks Pipelines API from a notebook and changed the cluster configuration to use a VM family that had available quota in my Azure subscription. After restarting the gateway, it provisioned successfully and the pipeline moved past the WAITING_FOR_RESOURCES state.

The original issue was caused by the gateway attempting to use an EDv4 VM family for which my subscription had no available quota.

Thanks again for your help!
Allan Solomon Mejia 2,595 Reputation points

2026-08-07T10:46:05.5766667+00:00

Hello @Naga Vardhan Sankar Raparthi

Please "Accept the Answer" if this information helped you. This will help us and others in the community as well.

1 answer

Your answer

Naga Vardhan Sankar Raparthi 0 Reputation points

2026-08-07T10:39:04.1233333+00:00

Thank you for the detailed explanation and guidance.

I was able to resolve the issue. I updated the ingestion gateway using the Databricks Pipelines API from a notebook and changed the cluster configuration to use a VM family that had available quota in my Azure subscription. After restarting the gateway, it provisioned successfully and the pipeline moved past the WAITING_FOR_RESOURCES state.

The original issue was caused by the gateway attempting to use an EDv4 VM family for which my subscription had no available quota.

Thanks again for your help!
Allan Solomon Mejia 2,595 Reputation points

2026-08-07T10:46:05.5766667+00:00

Hello @Naga Vardhan Sankar Raparthi

Please "Accept the Answer" if this information helped you. This will help us and others in the community as well.

Answer 1

Hi @Naga Vardhan Sankar Raparthi

Based on the information you've shared, this appears to be a limitation of the pipeline-managed Classic Compute rather than an issue with your Azure SQL Database or workspace configuration.

Since the CDC Gateway is managed by Databricks, the VM family is selected automatically. If the service is still attempting to provision Standard EDv4 instances, and your subscription has an EDv4 core quota of 0 (with quota requests unavailable because the SKU is retired), the gateway won't be able to start and will remain in WAITING_FOR_RESOURCES.

A few things to confirm:

Verify whether Serverless or Pro/Standard Compute is supported for your Lakeflow Connect CDC scenario, as these may avoid the legacy Classic Compute dependency.
If Classic Compute is currently required, there is no customer-facing option to override the managed gateway's VM family.
Since you've reproduced the issue across multiple regions and confirmed that other VM families have available quota, this points to the managed service requesting an unavailable SKU rather than a workspace configuration issue.

At this point, I'd recommend opening a Databricks Support case (or Azure Support if using Azure Databricks) and providing:

Workspace ID and region
Pipeline ID
Cluster event logs showing the QuotaExceeded error
Confirmation that the EDv4 quota is 0 and cannot be increased because the family is retired

They should be able to confirm whether this is a known issue with the current Lakeflow Connect CDC managed gateway provisioning logic or if there is a backend configuration or feature flag to use a supported VM family (such as EDv5/Esv5).

Please "Accept the Answer" if this information helped you. This will help us and others in the community as well.

Naga Vardhan Sankar Raparthi 0 Reputation points

2026-08-07T10:38:27.1866667+00:00

Thank you for your response. I was able to resolve the issue without opening a support case. The root cause was indeed the gateway compute configuration. I updated the ingestion gateway through the Databricks Pipelines API to use a VM family with available Azure quota, and the gateway started successfully.