Difference between Azure Event Hub and Azure Storage

Question

We need to stream events from Microsoft defender to Azure. For that 2 options are available: 1)Azure Storage 2)Event Hub
Now, I want to know the main difference between Azure storage and Event Hub as both are used to store events of Streaming API.
So that we can identify that how one option is better than another for forwarding events from Microsoft Defender Streaming APIs to Azure.

Answer 1

@Nirali Shah Welcome to the Microsoft Q&A Forum, Thank you for posting your query here!

Azure Event Hubs enables you to automatically capture the streaming data in Event Hubs in an Azure Blob storage or ADLS Gen 2 account of your choice, with the added flexibility of specifying a time or size interval. Setting up Capture is fast, there are no administrative costs to run it, and it scales automatically with Event Hubs throughput units in the standard tier or processing units in the premium tier. Event Hubs Capture is the easiest way to load streaming data into Azure, and enables you to focus on data processing rather than on data capture.

It has the following characteristics:

• Low latency
• Can receive and process millions of events per second
• At least once delivery of an event

For more information, see Event Hubs overview.

How Event Hubs Capture works
Event Hubs is a time-retention durable buffer for telemetry ingress, similar to a distributed log. The key to scaling in Event Hubs is the partitioned consumer model. Each partition is an independent segment of data and is consumed independently. Over time this data ages off, based on the configurable retention period. As a result, a given event hub never gets "too full."

Event Hubs Capture enables you to specify your own Azure Blob storage account and container, or Azure Data Lake Storage account, which are used to store the captured data. These accounts can be in the same region as your event hub or in another region, adding to the flexibility of the Event Hubs Capture feature.

Captured data is written in Apache Avro format: a compact, fast, binary format that provides rich data structures with inline schema. This format is widely used in the Hadoop ecosystem, Stream Analytics, and Azure Data Factory. More information about working with Avro is available later in this article.

Note: When you use no code editor in the Azure portal, you can capture streaming data in Event Hubs in an Azure Data Lake Storage Gen2 account in the Parquet format. For more information, see How to: capture data from Event Hubs in Parquet format and Tutorial: capture Event Hubs data in Parquet format and analyze with Azure Synapse Analytics.

Azure Event Hubs is a big data streaming platform and event ingestion service. It can receive and process millions of events per second. It facilitates the capture, retention, and replay of telemetry and event stream data. The data can come from many concurrent sources. Event Hubs allows telemetry and event data to be made available to various stream-processing infrastructures and analytics services. It's available either as data streams or bundled event batches. This service provides a single solution that enables rapid data retrieval for real-time processing, and repeated replay of stored raw data. It can capture the streaming data into a file for processing and analysis.

Capture uses a storage account you provide when enabled on an event hub. As it is your storage account, any changes for this configuration are billed to your Azure subscription.

• Azure Storage events allow applications to react to events, such as the creation and deletion of blobs. It does so without the need for complicated code or expensive and inefficient polling services. The best part is you only pay for what you use.

Blob storage events are pushed using Azure Event Grid to subscribers such as Azure Functions, Azure Logic Apps, or even to your own http listener. Event Grid provides reliable event delivery to your applications through rich retry policies and dead-lettering.

See the Blob storage events schema article to view the full list of the events that Blob storage supports.

Common Blob storage event scenarios include image or video processing, search indexing, or any file-oriented workflow. Asynchronous file uploads are a great fit for events. When changes are infrequent, but your scenario requires immediate responsiveness, event-based architecture can be especially efficient.

For more information, see this article.

Azure security baseline for Event Hubs

Azure security baseline for Azure Storage

Does Event Hub needs Azure Storage account?
Enable capturing of events streaming through Azure Event Hubs

• Azure Event Hubs provides encryption of data at rest with Azure Storage Service Encryption (Azure SSE). The Event Hubs service uses Azure Storage to store the data. All the data that's stored with Azure Storage is encrypted using Microsoft-managed keys. If you use your own key (also referred to as Bring Your Own Key (BYOK) or customer-managed key), the data is still encrypted using the Microsoft-managed key. For more information, refer to this article

Please let us know if you have any further queries. I’m happy to assist you further.

----------

Please do not forget to and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Hello @Nirali Shah

By default Microsoft Defender for Endpoint stores Endpoint events in Defender for Endpoint for the configured retention period; Max: 180 days.
So for longer data retention it is possible to export events to external sources like Azure Storage or Azure Event Hubs.

Event Hubs is used for streaming data to external systems such as third-party SIEMs or other Log Analytics solutions.

Storage account is useful for audit, static analysis, or backup. Compared to using Event Hubs, Storage is less expensive, logs can be kept there indefinitely and it also can be integrated with Event processing services.

Here is general example for Logs streaming sources:

If you think your question has been answered, click "Mark as Accept Answer" if just helped click "Vote as helpful". This can be beneficial to other community members reading this forum thread.