Send Synapse pyspark app logs to Geneva in a cost-effective and SFI Compliant way

Question

Send Synapse pyspark app logs to Geneva in a cost-effective and SFI Compliant way

Muhesh Krishnamurthy 0 Microsoft Employee

So currently we have use case where we have to send pyspark app logs to Geneva in a SFI Compliant way.

Our Current Architecture is Leveraging Synapse Diagnostic Emitters which offers 3 destinations

Azure Log Analytics (Secret based auth)
Storage Account (Secret based auth)
Event Hub (Secret based auth or Cert based auth)

We had to go with Event Hub since it is the only SFI compliant way possible out of the above 3 options, therefore our architecture became:
Synapse -> EventHub -> Event Trigger Azure Function (with ANTMDS Agent) -> Geneva

which works but it is not cost-effective.

Is there any SFI compliant and cost-effective way to send spark app logs to geneva from synapse.

Also is there a way to send Spark Application logs(Log4j logs) to Storage Account just with managed Identity ?

SAI JAGADEESH KUDIPUDI 2,790 Reputation points Microsoft External Staff Moderator

2026-03-17T07:04:46.2+00:00

Hi **Muhesh Krishnamurthy,
**Could you send the requested information in a private message?
SAI JAGADEESH KUDIPUDI 2,790 Reputation points Microsoft External Staff Moderator

2026-03-20T02:25:10.78+00:00
Hi Muhesh Krishnamurthy,
Event Hub remains mandatory to achieve SFI compliance in the current Synapse architecture. There is no direct Synapse-to-Geneva integration available today. The most effective optimization is to remove intermediate compute layers and reduce log volume while maintaining compliance.

Retain Event Hub for SFI compliance

Remove Azure Function to reduce cost

Use Geneva Agent for direct ingestion from Event Hub

Use Storage Account for low-cost long-term retention

Apply log filtering to minimize ingestion volume

Based on your requirement (SFI compliant + cost-effective + Synapse Spark logs → Geneva),

Current Architecture [Synapse → Event Hub → Azure Function → Geneva] This approach is expected and recommended for SFI compliance because:

Synapse diagnostic emitters support only:

Log Analytics, Storage Account, Event Hub.

Among these options:

Log Analytics and Storage rely on secret/key-based authentication

Event Hub supports certificate-based or Entra authentication (no secrets)

Event Hub is the only native SFI-compliant destination available in Synapse logging.
Why the Current Solution is Not Cost-Effective
The cost overhead is due to:

Event Hub ingestion and throughput units

Azure Function execution cost

Geneva ingestion pipeline overhead

Multiple processing layers increase overall cost and operational overhead.
Availability of a More Cost-Effective and SFI-Compliant Alternative

There is no direct native integration from Synapse to Geneva that satisfies:

SFI compliance, Managed Identity–only authentication, Cost efficiency

Because, Synapse does not support Geneva as a destination. Diagnostic emitters are limited to only three supported sinks

Recommended Architecture

Synapse → Event Hub (Entra/Certificate) → Geneva Agent

Remove Azure Function layer

Use Geneva agent to directly consume from Event Hub

Eliminates compute cost (Azure Function)

Maintains SFI compliance

Simplifies architecture

Sending Spark Logs to Storage Using Managed Identity

Synapse diagnostic emitter for Storage requires configuration via linked service.
Pure Managed Identity–only configuration is not natively supported
Use Linked Service configured with Managed Identity
This is an indirect use of Managed Identity
Not a fully native emitter-based MSI-only solution

Log4j Consideration

Synapse captures logs using diagnostic emitters, not direct Log4j appenders
Log filtering can be done using Log4j logger names
Full customization of Log4j pipelines (as seen in Databricks) is not supported

Reference Links:
Synapse Spark logs to Event Hub (official doc)
Certificate-based secure logging to Event Hub
Synapse Spark logs to Storage account

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.

Please do not forget to "up-vote" wherever the information provided helps you, as this can be beneficial to other community members.
SAI JAGADEESH KUDIPUDI 2,790 Reputation points Microsoft External Staff Moderator

2026-03-24T01:52:40.5+00:00

Hi **Muhesh Krishnamurthy,
**I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.

1 answer

Your answer

SAI JAGADEESH KUDIPUDI 2,790 Reputation points Microsoft External Staff Moderator

2026-03-17T07:04:46.2+00:00

Hi **Muhesh Krishnamurthy,
**Could you send the requested information in a private message?
SAI JAGADEESH KUDIPUDI 2,790 Reputation points Microsoft External Staff Moderator

2026-03-24T01:52:40.5+00:00

Hi **Muhesh Krishnamurthy,
**I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.

Answer 1

The available Synapse Apache Spark diagnostic emitters support only three destination types: Azure Log Analytics, Azure Storage, and Azure Event Hubs.

From the documented configuration options:

Azure Log Analytics: uses a workspace ID plus a secret (Log Analytics key) or a secret stored in Azure Key Vault.
Azure Storage: uses either an Access Key or SAS token, provided directly via secret or indirectly via Azure Key Vault.
Azure Event Hubs: uses a connection string provided directly via secret or indirectly via Azure Key Vault; there is also support for certificate-based service principal authentication for Event Hubs.

No documented option exists to:

Send Synapse Spark logs directly to Geneva from Synapse diagnostic emitters without going through one of these three destinations.
Use managed identity alone (without access key/SAS/Key Vault secret) to write Spark application logs (Log4j logs) from Synapse diagnostic emitters to Azure Storage.

For SFI-compliant routing, Event Hubs with secret or certificate-based authentication is the only documented path among the three emitter types. Any further cost optimization or Geneva integration would need to be implemented downstream of one of these supported destinations (for example, after Event Hubs), as there is no additional built-in, lower-cost, SFI-compliant Geneva target in the diagnostic emitter framework.

Similarly, for Storage, the documented authentication modes are AccessKey or SAS, either inline or via Key Vault; managed identity is not listed as a supported auth mechanism for the AzureStorage diagnostic emitter.

References:

Share via

Send Synapse pyspark app logs to Geneva in a cost-effective and SFI Compliant way

1 answer

Your answer