Azure Data Explorer interactive analytics

Data Explorer
Data Factory
Event Hubs
IoT Hub
Storage

Solution ideas

This article is a solution idea. If you'd like us to expand the content with more information, such as potential use cases, alternative services, implementation considerations, or pricing guidance, let us know by providing GitHub feedback.

This solution idea demonstrates interactive analytics with Azure Data Explorer to explore data with improvised, interactive, and lightning fast queries over small to extremely large volumes of data. This data exploration can be done using native Azure Data Explorer tools or alternative tools of your choice. This solution focuses on the integration of Azure Data Explorer with rest of the data platform ecosystem.

Potential use cases

This solution is already being used by Microsoft customers to track user activity, manage user profiles and user segmentation scenarios. For example, global software company Episerver uses Azure Data Explorer to gain enhanced consumer insight.

Architecture

Interactive analytics with Azure Data Explorer

Dataflow

  1. Raw structured, semi-structured, and unstructured (free text) data such as, any type of logs, business events, and user activities can be ingested into Azure Data Explorer from various sources. Ingest the data in streaming or batch mode using various methods.
  2. Ingest data into Azure Data Explorer with low-latency and high-throughput using its connectors for Azure Data Factory, Azure Event Hub, Azure IoT Hub, Kafka, and so on. Instead, ingest data through Azure Storage (Blob or ADLS Gen2), which uses Azure Event Grid and triggers the ingestion pipeline to Azure Data Explorer. You can also continuously export data to Azure Storage in compressed, partitioned parquet format and seamlessly query that data as detailed in continuous data export overview.
  3. Run interactive queries over small to extremely large volumes of data using native Azure Data Explorer tools or alternative tools of your choice. Azure Data Explorer provides many plugins and integrations with the rest of the data platform ecosystem. Use any of the following tools and integrations:
  4. Enrich data running federated queries by combining data from SQL database and Azure Cosmos DB using Azure Data Explorer plugins.

Components

  • Azure Event Hub: Fully managed, real-time data ingestion service that's simple, trusted, and scalable.
  • Azure IoT Hub: Managed service to enable bi-directional communication between IoT devices and Azure.
  • Kafka on HDInsight: Easy, cost-effective, enterprise-grade service for open-source analytics with Apache Kafka.
  • Azure Data Factory: Hybrid data integration service that simplifies ETL at scale.
  • Azure Data Explorer: Fast, fully managed and highly scalable data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more.
  • Azure Data Explorer Dashboards: Natively export Kusto queries that were explored in the Web UI to optimized dashboards.
  • Azure Cosmos DB: Fully managed fast NoSQL database service for modern app development with open APIs for any scale.
  • Azure SQL DB: Build apps that scale with the pace of your business with managed and intelligent SQL in the cloud.

Contributors

This article is maintained by Microsoft. It was originally written by the following contributors.

Principal author:

Next steps

For more information, see Azure Data Explorer documentation.