Edit

Azure Data Explorer interactive analytics

Azure Data Explorer
Azure Data Factory
Azure Event Hubs
Azure IoT Hub
Azure Storage

Solution ideas

This article is a solution idea. If you'd like us to expand the content with more information, such as potential use cases, alternative services, implementation considerations, or pricing guidance, let us know by providing GitHub feedback.

This solution idea demonstrates how to use interactive analytics in Azure Data Explorer. It describes how you can examine structured, semi-structured, and unstructured data with improvised, interactive, fast queries.

Jupyter is a trademark of its respective company. No endorsement is implied by the use of this mark. Apache® and Apache Kafka® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.

Architecture

Interactive analytics with Azure Data Explorer.

Download a Visio file of this architecture.

Dataflow

  1. Raw structured, semi-structured, and unstructured (free text) data such as, any type of logs, business events, and user activities can be ingested into Azure Data Explorer from various sources. Ingest the data in streaming or batch mode using various methods.
  2. Ingest data into Azure Data Explorer with low-latency and high-throughput using its connectors for Azure Data Factory, Azure Event Hubs, Azure IoT Hub, Kafka, and so on. Instead, ingest data through Azure Storage (Blob or ADLS Gen2), which uses Azure Event Grid and triggers the ingestion pipeline to Azure Data Explorer. You can also continuously export data to Azure Storage in compressed, partitioned parquet format and seamlessly query that data as detailed in continuous data export overview.
  3. Run interactive queries over small to extremely large volumes of data using native Azure Data Explorer tools or alternative tools of your choice. Azure Data Explorer provides many plugins and integrations with the rest of the data platform ecosystem. Use any of the following tools and integrations:
  4. Enrich data running federated queries by combining data from SQL database and Azure Cosmos DB using Azure Data Explorer plugins.

Components

  • Azure Event Hubs: Fully managed, real-time data ingestion service that's simple, trusted, and scalable.
  • Azure IoT Hub: Managed service to enable bi-directional communication between IoT devices and Azure.
  • Kafka on HDInsight: Easy, cost-effective, enterprise-grade service for open-source analytics with Apache Kafka.
  • Azure Data Factory: Hybrid data integration service that simplifies ETL at scale.
  • Azure Data Explorer: Fast, fully managed and highly scalable data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more.
  • Azure Data Explorer Dashboards: Natively export Kusto queries that were explored in the Web UI to optimized dashboards.
  • Azure Cosmos DB: Fully managed fast NoSQL database service for modern app development with open APIs for any scale.
  • Azure SQL DB: Build apps that scale with the pace of your business with managed and intelligent SQL in the cloud.

Scenario details

This solution idea demonstrates how to use interactive analytics with Azure Data Explorer to explore data with improvised, interactive, and fast queries over small to extremely large volumes of data. This data exploration can be done using native Azure Data Explorer tools or alternative tools of your choice. This solution focuses on the integration of Azure Data Explorer with rest of the data platform ecosystem.

Potential use cases

This solution is used by Microsoft customers to track user activity, manage user profiles and user segmentation scenarios.

Contributors

This article is maintained by Microsoft. It was originally written by the following contributors.

Principal author:

Next steps

For more information, see Azure Data Explorer documentation.