Ports used by Apache Hadoop services on HDInsight

This document provides a list of the ports used by Apache Hadoop services running on HDInsight clusters. It also provides information on ports used to connect to the cluster using SSH.

Public ports vs. non-public ports

Linux-based HDInsight clusters only expose three ports publicly on the internet: 22, 23, and 443. These ports secure cluster access using SSH and services exposed over the secure HTTPS protocol.

HDInsight is implemented by several Azure Virtual Machines (cluster nodes) running on an Azure Virtual Network. From within the virtual network, you can access ports not exposed over the internet. If you connect via SSH to the head node, you can directly access services running on the cluster nodes.

Important

If you do not specify an Azure Virtual Network as a configuration option for HDInsight, one is created automatically. However, you can't join other machines (such as other Azure Virtual Machines or your client development machine) to this virtual network.

To join additional machines to the virtual network, you must create the virtual network first, and then specify it when creating your HDInsight cluster. For more information, see Plan a virtual network for HDInsight.

Public ports

All the nodes in an HDInsight cluster are located in an Azure Virtual Network. The nodes can't be directly accessed from the internet. A public gateway provides internet access to the following ports, which are common across all HDInsight cluster types.

Service Port Protocol Description
sshd 22 SSH Connects clients to sshd on the primary headnode. For more information, see Use SSH with HDInsight.
sshd 22 SSH Connects clients to sshd on the edge node. For more information, see Use SSH with HDInsight.
sshd 23 SSH Connects clients to sshd on the secondary headnode. For more information, see Use SSH with HDInsight.
Ambari 443 HTTPS Ambari web UI. See Manage HDInsight using the Apache Ambari Web UI
Ambari 443 HTTPS Ambari REST API. See Manage HDInsight using the Apache Ambari REST API
WebHCat 443 HTTPS HCatalog REST API. See Use MapReduce with Curl
HiveServer2 443 ODBC Connects to Hive using ODBC. See Connect Excel to HDInsight with the Microsoft ODBC driver.
HiveServer2 443 JDBC Connects to ApacheHive using JDBC. See Connect to Apache Hive on HDInsight using the Hive JDBC driver

The following are available for specific cluster types:

Service Port Protocol Cluster type Description
Stargate 443 HTTPS HBase HBase REST API. See Get started using Apache HBase
Livy 443 HTTPS Spark Spark REST API. See Submit Apache Spark jobs remotely using Apache Livy
Spark Thrift server 443 HTTPS Spark Spark Thrift server used to submit Hive queries. See Use Beeline with Apache Hive on HDInsight
Kafka REST proxy 443 HTTPS Kafka Kafka REST API. See Interact with Apache Kafka clusters in Azure HDInsight using a REST proxy

Authentication

All services publicly exposed on the internet must be authenticated:

Port Credentials
22 or 23 The SSH user credentials specified during cluster creation
443 The login name (default: admin) and password that were set during cluster creation

Non-public ports

Note

Some services are only available on specific cluster types. For example, HBase is only available on HBase cluster types.

Important

Some services only run on one headnode at a time. If you attempt to connect to the service on the primary headnode and receive an error, retry using the secondary headnode.

Ambari

Service Nodes Port URL path Protocol
Ambari web UI Head nodes 8080 / HTTP
Ambari REST API Head nodes 8080 /api/v1 HTTP

Examples:

  • Ambari REST API: curl -u admin "http://10.0.0.11:8080/api/v1/clusters"

HDFS ports

Service Nodes Port Protocol Description
NameNode web UI Head nodes 30070 HTTPS Web UI to view status
NameNode metadata service head nodes 8020 IPC File system metadata
DataNode All worker nodes 30075 HTTPS Web UI to view status, logs, and so on.
DataNode All worker nodes 30010   Data transfer
DataNode All worker nodes 30020 IPC Metadata operations
Secondary NameNode Head nodes 50090 HTTP Checkpoint for NameNode metadata

YARN ports

Service Nodes Port Protocol Description
Resource Manager web UI Head nodes 8088 HTTP Web UI for Resource Manager
Resource Manager web UI Head nodes 8090 HTTPS Web UI for Resource Manager
Resource Manager admin interface head nodes 8141 IPC For application submissions (Hive, Hive server, Pig, and so on.)
Resource Manager scheduler head nodes 8030 HTTP Administrative interface
Resource Manager application interface head nodes 8050 HTTP Address of the applications manager interface
NodeManager All worker nodes 30050   The address of the container manager
NodeManager web UI All worker nodes 30060 HTTP Resource Manager interface
Timeline address Head nodes 10200 RPC The Timeline service RPC service.
Timeline web UI Head nodes 8188 HTTP The Timeline service web UI

Hive ports

Service Nodes Port Protocol Description
HiveServer2 Head nodes 10001 Thrift Service for connecting to Hive (Thrift/JDBC)
Hive Metastore Head nodes 9083 Thrift Service for connecting to Hive metadata (Thrift/JDBC)

WebHCat ports

Service Nodes Port Protocol Description
WebHCat server Head nodes 30111 HTTP Web API on top of HCatalog and other Hadoop services

MapReduce ports

Service Nodes Port Protocol Description
JobHistory Head nodes 19888 HTTP MapReduce JobHistory web UI
JobHistory Head nodes 10020   MapReduce JobHistory server
ShuffleHandler   13562   Transfers intermediate Map outputs to requesting Reducers

Oozie

Service Nodes Port Protocol Description
Oozie server Head nodes 11000 HTTP URL for Oozie service
Oozie server Head nodes 11001 HTTP Port for Oozie admin

Ambari Metrics

Service Nodes Port Protocol Description
TimeLine (Application history) Head nodes 6188 HTTP The TimeLine service web UI
TimeLine (Application history) Head nodes 30200 RPC The TimeLine service web UI

HBase ports

Service Nodes Port Protocol Description
HMaster Head nodes 16000    
HMaster info Web UI Head nodes 16010 HTTP The port for the HBase Master web UI
Region server All worker nodes 16020  
Region server info Web UI   All worker nodes 16030 HTTP The port for the HBase Region server Web UI
2181 The port that clients use to connect to ZooKeeper

Kafka ports

Service Nodes Port Protocol Description
Broker Worker nodes 9092 Kafka Wire Protocol Used for client communication
  Zookeeper nodes 2181   The port that clients use to connect to Zookeeper
REST proxy Kafka management nodes 9400 HTTPS Kafka REST specification

Spark ports

Service Nodes Port Protocol URL path Description
Spark Thrift servers Head nodes 10002 Thrift   Service for connecting to Spark SQL (Thrift/JDBC)
Livy server Head nodes 8998 HTTP   Service for running statements, jobs, and applications
Jupyter Notebook Head nodes 8001 HTTP   Jupyter Notebook website

Examples:

  • Livy: curl -u admin -G "http://10.0.0.11:8998/". In this example, 10.0.0.11 is the IP address of the headnode that hosts the Livy service.