Azure HDInsight documentation

Azure HDInsight is a managed Apache Hadoop service that lets you run Apache Spark, Apache Hive, Apache Kafka, Apache HBase, and more in the cloud.

About HDInsight

Overview

• What is Azure HDInsight?
• HDInsight OSS components and versions

Get started

• Build analytical solutions with Azure HDInsight
• Create HDInsight clusters
• Monitoring with HDInsight
• Scale cluster to save cost
• HDInsight security overview

Apache Spark

Get started

• What is Apache Spark?
• Create Spark clusters and run Spark in Jupyter
• Load data and run Spark queries

How-To Guide

• Manage Spark dependencies
• Optimize Spark jobs

Apache Hadoop

Get started

• What is Apache Hadoop?
• Create Hadoop clusters and run Hive queries
• Run Map Reduce samples

How-To Guide

• Monitor and manage Hadoop clusters
• Use JDBC/ODBC driver

Integration

How-To Guide

• Spark/Hive - Connect Spark and Hive with Hive Warehouse connector
• Spark/Kafka - Apache Spark structured streaming with Apache Kafka
• Spark/HBase - Query Apache HBase with Apache Spark
• Create on-demand clusters using ADF

Concept

• Selecting the right VM size
• HDInsight supported VM types
• Compare storage options

Apache Kafka

Get started

• What is Apache Kafka?
• Create Kafka clusters and manage Kafka topics
• Use producer and consumer APIs

How-To Guide

• Use Kafka REST proxy
• Kafka TLS encryption & authentication

Interactive query

Get started

• What is Interactive Query?
• Analyze flight data with Apache Hive

How-To Guide

• Cluster sizing guide
• Connect Hive with Power BI

Enterprise readiness

Get started

• HDInsight security overview
• Plan virtual network
• Hadoop on Active Directory: Enterprise Security Package
• Migrate on-premises clusters to the cloud

Apache HBase

Get started

• What is Apache HBase?
• Use HBase in Azure HDInsight
• Query HBase with Apache Phoenix

How-To Guide

• Enable Accelerated Writes for HBase
• Apache Phoenix best practices