Use NSG to restrict traffic to HDInsight on AKS
Note
We will retire Azure HDInsight on AKS on January 31, 2025. Before January 31, 2025, you will need to migrate your workloads to Microsoft Fabric or an equivalent Azure product to avoid abrupt termination of your workloads. The remaining clusters on your subscription will be stopped and removed from the host.
Only basic support will be available until the retirement date.
Important
This feature is currently in preview. The Supplemental Terms of Use for Microsoft Azure Previews include more legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability. For information about this specific preview, see Azure HDInsight on AKS preview information. For questions or feature suggestions, please submit a request on AskHDInsight with the details and follow us for more updates on Azure HDInsight Community.
HDInsight on AKS relies on AKS outbound dependencies and they're entirely defined with FQDNs, which don't have static addresses behind them. The lack of static IP addresses means one can't use Network Security Groups (NSGs) to lock down the outbound traffic from the cluster using IPs.
If you still prefer to use NSG to secure your traffic, then you need to configure the following rules in the NSG to do a coarse-grained control.
Learn how to create a security rule in NSG.
Outbound security rules (Egress traffic)
Common traffic
| Destination | Destination Endpoint | Protocol | Port |
|---|---|---|---|
| Service Tag | AzureCloud.<Region> |
UDP | 1194 |
| Service Tag | AzureCloud.<Region> |
TCP | 9000 |
| Any | * | TCP | 443, 80 |
Cluster specific traffic
This section outlines cluster specific traffic that an enterprise can apply.
Trino
| Destination | Destination Endpoint | Protocol | Port |
|---|---|---|---|
| Any | * | TCP | 1433 |
| Service Tag | Sql.<Region> |
TCP | 11000-11999 |
Spark
| Destination | Destination Endpoint | Protocol | Port |
|---|---|---|---|
| Any | * | TCP | 1433 |
| Service Tag | Sql.<Region> |
TCP | 11000-11999 |
| Service Tag | Storage.<Region> |
TCP | 445 |
Apache Flink
None
Inbound security rules (Ingress traffic)
When clusters are created, then certain ingress public IPs also get created. To allow requests to be sent to the cluster, you need to allowlist the traffic to these public IPs with port 80 and 443.
The following Azure CLI command can help you get the ingress public IP:
aksManagedResourceGroup="az rest --uri https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.HDInsight/clusterpools/{clusterPoolName}\?api-version\=2023-06-01-preview --query properties.managedResourceGroupName -o tsv --query properties.aksManagedResourceGroupName -o tsv"
az network public-ip list --resource-group $aksManagedResourceGroup --query "[?starts_with(name, 'kubernetes')].{Name:name, IngressPublicIP:ipAddress}" --output table
| Source | Source IP addresses/CIDR ranges | Protocol | Port |
|---|---|---|---|
| IP Addresses | <Public IP retrieved from above command> |
TCP | 80 |
| IP Addresses | <Public IP retrieved from above command> |
TCP | 443 |