How do you make streaming data faster

Samy Abdul 3,366 Reputation points
2021-08-28T07:51:29.633+00:00

Hi All, what are the key considerations make ingestion of streaming data faster. what is the architecture design for it by using event hub and storing in ADLS. Thanks.

Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
580 questions
Azure Event Grid
Azure Event Grid
An Azure event routing service designed for high availability, consistent performance, and dynamic scale.
337 questions
0 comments No comments
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 82,271 Reputation points Microsoft Employee
    2021-08-30T09:16:55.547+00:00

    Hello @Samy Abdul ,

    Thanks for the question and using MS Q&A platform.

    In the question doesn't have details about you business requirement and sharing the solution as per the best practices.

    Event Hubs is a fully managed Platform-as-a-Service (PaaS) with little configuration or management overhead, so you focus on your business solutions.

    Event Hubs for Apache Kafka ecosystems gives you the PaaS Kafka experience without having to manage, configure, or run your clusters.

    If you are looking to stream data from event Hubs and then store data in ADLS?

    Azure Event Hubs enables you to automatically capture the streaming data in Event Hubs in an Azure Blob storage or Azure Data Lake Storage Gen 1 or Gen 2 account of your choice, with the added flexibility of specifying a time or size interval. Setting up Capture is fast, there are no administrative costs to run it, and it scales automatically with Event Hubs throughput units in the standard tier or processing units in the premium tier. Event Hubs Capture is the easiest way to load streaming data into Azure, and enables you to focus on data processing rather than on data capture.

    127546-image.png

    For more details, refer to Capture events through Azure Event Hubs in Azure Blob Storage or Azure Data Lake Storage

    Azure Event Grid allows you to easily build applications with event-based architectures. First, select the Azure resource you would like to subscribe to, and then give the event handler or WebHook endpoint to send the event to. Event Grid has built-in support for events coming from Azure services, like storage blobs and resource groups. Event Grid also has support for your own events, using custom topics.

    127566-image.png

    Azure Event Grid is an intelligent event routing service that enables you to react to notifications or events from apps and services. For example, it can trigger an Azure Function to process Event Hubs data that's captured to a Blob storage or Data Lake Storage. The below image shows you how to use Event Grid and Azure Functions to migrate captured Event Hubs data from blob storage to Azure Synapse Analytics, specifically a dedicated SQL pool.

    127573-image.png

    For more details, refer to Tutorial: Stream big data into a data warehouse

    What are the key considerations make ingestion of streaming data faster?

    Build an end-to-end serverless streaming platform with Event Hubs and Stream Analytics

    127583-image.png

    This reference architecture shows a serverless, event-driven architecture that ingests a stream of data, processes the data, and writes the results to a back-end database.

    Different streaming tutorials:

    Hope this helps. Do let us know if you any further queries.

    ---------------------------------------------------------------------------

    Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

    0 comments No comments

0 additional answers

Sort by: Most helpful