Share via

How to access EventHub while using Azure Synapse spark pool?

Guilherme Contini 20 Reputation points Microsoft Employee
2026-01-05T19:33:01.3733333+00:00

Hi all — I’m trying to route Synapse Spark diagnostics to Event Hubs → Capture → ADX using the AzureEventHub diagnostic emitter. After disabling local auth (per Safe Secrets guidance), the emitter stops sending messages.

Here’s what I observe:

  • Event Hub metrics show Incoming Requests increasing, all successful

But Incoming Messages = 0 during every Spark job

Manual test from the Azure Portal does send messages (so Event Hub is healthy)

ADX capture files are 0 bytes, ingestion fails with BadRequest_NoRecordsOrWrongFormat

But MSI token acquisition from Spark hangs forever and IMDS is unreachable

Tried AAD app with secrets → unsupported by emitter (no messages)

Tried certificate-based auth → Entra policy blocks all certificates (maxLifetime = PT0S)

Question: Given that:

SAS is disabled (LocalAuth=false)

MSI cannot work (IMDS blocked)

Certificate auth is forbidden by tenant policy

SPN secrets are not supported by the diagnostic emitter

→ What is the recommended supported authentication method for Synapse Spark diagnostic emitter in a tenant where MI is blocked and certificate credentials are disallowed?

Thanks in advance — I’m trying to restore Synapse → Event Hub → ADX diagnostic flow in a compliant way.Hi all — I’m trying to route Synapse Spark diagnostics to Event Hubs → Capture → ADX using the AzureEventHub diagnostic emitter. After disabling local auth (per Safe Secrets guidance), the emitter stops sending messages.

Here’s what I observe:

Event Hub metrics show Incoming Requests increasing, all successful

But Incoming Messages = 0 during every Spark job

Manual test from the Azure Portal does send messages (so Event Hub is healthy)

ADX capture files are 0 bytes, ingestion fails with BadRequest_NoRecordsOrWrongFormat

But MSI token acquisition from Spark hangs forever and IMDS is unreachable

Tried AAD app with secrets → unsupported by emitter (no messages)

Tried certificate-based auth → Entra policy blocks all certificates (maxLifetime = PT0S)

Question:
Given that:

SAS is disabled (LocalAuth=false)

MSI cannot work (IMDS blocked)

Certificate auth is forbidden by tenant policy

SPN secrets are not supported by the diagnostic emitter

→ What is the recommended supported authentication method for Synapse Spark diagnostic emitter in a tenant where MI is blocked and certificate credentials are disallowed?

Thanks in advance — I’m trying to restore Synapse → Event Hub → ADX diagnostic flow in a compliant way.

Azure Event Hubs

Answer accepted by question author

Anonymous
2026-01-05T19:45:52.0866667+00:00

Hi @Guilherme Contini

Thank you for contacting Microsoft Q&A. Please find below the detailed steps to address the reported issue

Authoritative Authentication Requirements

The emitter supports only the following authentication modes:

SAS Connection String (LocalAuth / Shared Access Key)

Documented here: “spark.synapse.diagnostic.emitter.<dest>.secret <connection-string>” [Collect yo...soft Learn | Learn.Microsoft.com]

Managed Identity (MSI)

Required for Key Vault secret retrieval and for Event Hub access when using passwordless auth. [Synapse Sp...UTHBEARER) | IcM]

Certificate‑based Service Principal Authentication

Full official setup: [Collect Ap...soft Learn | Learn.Microsoft.com]

➡ NO OTHER AUTH METHODS ARE SUPPORTED.

Methods That Will NOT Work (Confirmed)

1. AAD App + Client Secret

Explicitly unsupported by the AzureEventHub diagnostic emitter. [learn.microsoft.com]

2. Certificate‑based Auth

Supported by the emitter, but blocked by your tenant’s Entra ID certificate policy. [learn.microsoft.com]

3. Managed Identity

Unavailable due to IMDS unreachable → token acquisition hangs indefinitely. [learn.microsoft.com]

THE ONLY COMPLIANT & SUPPORTED RESOLUTION

Option 1 (Recommended): Re‑enable SAS (LocalAuth=true) ONLY for the diagnostic emitter

This is Microsoft’s supported and intended fallback when MSI/cert auth cannot be used.

Why this is justified:

SAS (connection string) is an officially supported configuration parameter of the diagnostic emitter. [Collect yo...soft Learn | Learn.Microsoft.com]

Azure Event Hubs Data Sender role is not required when using SAS.

SAS keys can be scoped to a single Event Hub + “Send” permission → minimal blast radius.

Meets Safe Secrets requirements if stored in Azure Key Vault and referenced via:

spark.synapse.diagnostic.emitter.MyDestination1.secret.keyVault <linked-service>

spark.synapse.diagnostic.emitter.MyDestination1.secret.keyVault.secretName <secretName>

(Supported by emitter → verified from Microsoft Q&A examples) [Spark Diag...rosoft Q&A | Learn.Microsoft.com]

Why this is the only viable auth:

All other supported methods are blocked by the customer's environment.

Option 2: Unblock MSI (Preferred from an Azure security standpoint)

If the customer can update network rules so Spark runtime can reach IMDS (169.254.169.254), Managed Identity will work exactly as designed.

Official reference: IMDS unreachable → MSI unavailable. [learn.microsoft.com]

This restores full passwordless authentication.

Option 3: Unblock certificate-based auth

If security policy can be amended to permit certificate lifetimes > 0s, the official Microsoft workflow works end‑to‑end. [Collect Ap...soft Learn | Learn.Microsoft.com]

Was this answer helpful?

0 comments

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest

  1. AI answer

    2026-01-05T19:34:11.96+00:00

    To access Azure Event Hubs while using Azure Synapse Spark pool, you can utilize the Synapse Apache Spark diagnostic emitter extension. However, given your constraints regarding authentication methods, the recommended approach would be to use Azure Key Vault to store your connection string securely and reference it in your Spark configuration.

    Here’s a summary of how you can set this up:

    1. Use Azure Key Vault: Store your Event Hub connection string in Azure Key Vault. This allows you to manage secrets securely and comply with your tenant's policies.
    2. Configure Spark Diagnostic Emitter: In your Apache Spark configuration file, specify the connection string using the Key Vault parameters:
      • spark.synapse.diagnostic.emitter.<destination>.secret.keyVault - The name of your Azure Key Vault.
      • spark.synapse.diagnostic.emitter.<destination>.secret.keyVault.secretName - The name of the secret where your connection string is stored.
      • Ensure that your Synapse workspace has the necessary permissions to access the Key Vault.
    3. Upload Configuration: After creating your configuration file, upload it to your Synapse Spark pool via Synapse Studio.
    4. Check Permissions: Make sure that the managed identity (if used) has read permissions on the Key Vault to access the connection string.

    This method allows you to bypass the limitations you are facing with MSI and certificate-based authentication while still securely accessing Azure Event Hubs.

    References:

    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.