Rediger

Del via


Outbound network and FQDN rules for Azure Kubernetes Service (AKS) clusters

This article provides the necessary details that allow you to secure outbound traffic from your Azure Kubernetes Service (AKS). It contains the cluster requirements for a base AKS deployment and additional requirements for optional addons and features. You can apply this information to any outbound restriction method or appliance.

To see an example configuration using Azure Firewall, visit Control egress traffic using Azure Firewall in AKS.

Background

AKS clusters are deployed on a virtual network. This network can either be customized and pre-configured by you or it can be created and managed by AKS. In either case, the cluster has outbound, or egress, dependencies on services outside of the virtual network.

For management and operational purposes, nodes in an AKS cluster need to access certain ports and fully qualified domain names (FQDNs). These endpoints are required for the nodes to communicate with the API server or to download and install core Kubernetes cluster components and node security updates. For example, the cluster needs to pull base system container images from Microsoft Container Registry (MCR).

The AKS outbound dependencies are almost entirely defined with FQDNs, which don't have static addresses behind them. The lack of static addresses means you can't use network security groups (NSGs) to lock down the outbound traffic from an AKS cluster.

By default, AKS clusters have unrestricted outbound internet access. This level of network access allows nodes and services you run to access external resources as needed. If you wish to restrict egress traffic, a limited number of ports and addresses must be accessible to maintain healthy cluster maintenance tasks. The simplest solution to securing outbound addresses is using a firewall device that can control outbound traffic based on domain names. Azure Firewall can restrict outbound HTTP and HTTPS traffic based on the FQDN of the destination. You can also configure your preferred firewall and security rules to allow these required ports and addresses.

Important

This document covers only how to lock down the traffic leaving the AKS subnet. AKS has no ingress requirements by default. Blocking internal subnet traffic using network security groups (NSGs) and firewalls isn't supported. To control and block the traffic within the cluster, see Secure traffic between pods using network policies in AKS.

Required outbound network rules and FQDNs for AKS clusters

The following network and FQDN/application rules are required for an AKS cluster. You can use them if you wish to configure a solution other than Azure Firewall.

  • IP address dependencies are for non-HTTP/S traffic (both TCP and UDP traffic).
  • FQDN HTTP/HTTPS endpoints can be placed in your firewall device.
  • Wildcard HTTP/HTTPS endpoints are dependencies that can vary with your AKS cluster based on a number of qualifiers.
  • AKS uses an admission controller to inject the FQDN as an environment variable to all deployments under kube-system and gatekeeper-system. This ensures all system communication between nodes and API server uses the API server FQDN and not the API server IP. You can get the same behavior on your own pods, in any namespace, by annotating the pod spec with an annotation named kubernetes.azure.com/set-kube-service-host-fqdn. If that annotation is present, AKS will set the KUBERNETES_SERVICE_HOST variable to the domain name of the API server instead of the in-cluster service IP. This is useful in cases where the cluster egress is via a layer 7 firewall.
  • If you have an app or solution that needs to talk to the API server, you must either add an additional network rule to allow TCP communication to port 443 of your API server's IP OR , if you have a layer 7 firewall configured to allow traffic to the API Server's domain name, set kubernetes.azure.com/set-kube-service-host-fqdn in your pod specs.
  • On rare occasions, if there's a maintenance operation, your API server IP might change. Planned maintenance operations that can change the API server IP are always communicated in advance.
  • Under certain circumstances, it might happen that traffic towards "md-*.blob.storage.azure.net" is required. This dependency is due to some internal mechanisms of Azure Managed Disks. You might also want to use the Storage service tag.
  • You might notice traffic towards "umsa*.blob.core.windows.net" endpoint. This endpoint is used to store manifests for Azure Linux VM Agent & Extensions and is regularly checked to download new versions.

Azure Global required network rules

Destination Endpoint Protocol Port Use
*:1194
Or
ServiceTag - AzureCloud.<Region>:1194
Or
Regional CIDRs - RegionCIDRs:1194
Or
APIServerPublicIP:1194 (only known after cluster creation)
UDP 1194 For tunneled secure communication between the nodes and the control plane. This isn't required for private clusters, or for clusters with the konnectivity-agent enabled.
*:9000
Or
ServiceTag - AzureCloud.<Region>:9000
Or
Regional CIDRs - RegionCIDRs:9000
Or
APIServerPublicIP:9000 (only known after cluster creation)
TCP 9000 For tunneled secure communication between the nodes and the control plane. This isn't required for private clusters, or for clusters with the konnectivity-agent enabled.
*:123 or ntp.ubuntu.com:123 (if using Azure Firewall network rules) UDP 123 Required for Network Time Protocol (NTP) time synchronization on Linux nodes. This isn't required for nodes provisioned after March 2021.
CustomDNSIP:53 (if using custom DNS servers) UDP 53 If you're using custom DNS servers, you must ensure they're accessible by the cluster nodes.
APIServerPublicIP:443 (if running pods/deployments that access the API Server) TCP 443 Required if running pods/deployments that access the API Server, those pods/deployments would use the API IP. This port isn't required for private clusters.

Azure Global required FQDN / application rules

Destination FQDN Port Use
*.hcp.<location>.azmk8s.io HTTPS:443 Required for Node <-> API server communication. Replace <location> with the region where your AKS cluster is deployed. This is required for clusters with konnectivity-agent enabled. Konnectivity also uses Application-Layer Protocol Negotiation (ALPN) to communicate between agent and server. Blocking or rewriting the ALPN extension will cause a failure. This isn't required for private clusters.
mcr.microsoft.com HTTPS:443 Required to access images in Microsoft Container Registry (MCR). This registry contains first-party images/charts (for example, coreDNS, etc.). These images are required for the correct creation and functioning of the cluster, including scale and upgrade operations.
*.data.mcr.microsoft.com, mcr-0001.mcr-msedge.net HTTPS:443 Required for MCR storage backed by the Azure content delivery network (CDN).
management.azure.com HTTPS:443 Required for Kubernetes operations against the Azure API.
login.microsoftonline.com HTTPS:443 Required for Microsoft Entra authentication.
packages.microsoft.com HTTPS:443 This address is the Microsoft packages repository used for cached apt-get operations. Example packages include Moby, PowerShell, and Azure CLI.
acs-mirror.azureedge.net HTTPS:443 This address is for the repository required to download and install required binaries like kubenet and Azure CNI.

Microsoft Azure operated by 21Vianet required network rules

Destination Endpoint Protocol Port Use
*:1194
Or
ServiceTag - AzureCloud.Region:1194
Or
Regional CIDRs - RegionCIDRs:1194
Or
APIServerPublicIP:1194 (only known after cluster creation)
UDP 1194 For tunneled secure communication between the nodes and the control plane.
*:9000
Or
ServiceTag - AzureCloud.<Region>:9000
Or
Regional CIDRs - RegionCIDRs:9000
Or
APIServerPublicIP:9000 (only known after cluster creation)
TCP 9000 For tunneled secure communication between the nodes and the control plane.
*:22
Or
ServiceTag - AzureCloud.<Region>:22
Or
Regional CIDRs - RegionCIDRs:22
Or
APIServerPublicIP:22 (only known after cluster creation)
TCP 22 For tunneled secure communication between the nodes and the control plane.
*:123 or ntp.ubuntu.com:123 (if using Azure Firewall network rules) UDP 123 Required for Network Time Protocol (NTP) time synchronization on Linux nodes.
CustomDNSIP:53 (if using custom DNS servers) UDP 53 If you're using custom DNS servers, you must ensure they're accessible by the cluster nodes.
APIServerPublicIP:443 (if running pods/deployments that access the API Server) TCP 443 Required if running pods/deployments that access the API Server, those pod/deployments would use the API IP.

Microsoft Azure operated by 21Vianet required FQDN / application rules

Destination FQDN Port Use
*.hcp.<location>.cx.prod.service.azk8s.cn HTTPS:443 Required for Node <-> API server communication. Replace <location> with the region where your AKS cluster is deployed.
*.tun.<location>.cx.prod.service.azk8s.cn HTTPS:443 Required for Node <-> API server communication. Replace <location> with the region where your AKS cluster is deployed.
mcr.microsoft.com HTTPS:443 Required to access images in Microsoft Container Registry (MCR). This registry contains first-party images/charts (for example, coreDNS, etc.). These images are required for the correct creation and functioning of the cluster, including scale and upgrade operations.
.data.mcr.microsoft.com HTTPS:443 Required for MCR storage backed by the Azure Content Delivery Network (CDN).
management.chinacloudapi.cn HTTPS:443 Required for Kubernetes operations against the Azure API.
login.chinacloudapi.cn HTTPS:443 Required for Microsoft Entra authentication.
packages.microsoft.com HTTPS:443 This address is the Microsoft packages repository used for cached apt-get operations. Example packages include Moby, PowerShell, and Azure CLI.
*.azk8s.cn HTTPS:443 This address is for the repository required to download and install required binaries like kubenet and Azure CNI.

Azure US Government required network rules

Destination Endpoint Protocol Port Use
*:1194
Or
ServiceTag - AzureCloud.<Region>:1194
Or
Regional CIDRs - RegionCIDRs:1194
Or
APIServerPublicIP:1194 (only known after cluster creation)
UDP 1194 For tunneled secure communication between the nodes and the control plane.
*:9000
Or
ServiceTag - AzureCloud.<Region>:9000
Or
Regional CIDRs - RegionCIDRs:9000
Or
APIServerPublicIP:9000 (only known after cluster creation)
TCP 9000 For tunneled secure communication between the nodes and the control plane.
*:123 or ntp.ubuntu.com:123 (if using Azure Firewall network rules) UDP 123 Required for Network Time Protocol (NTP) time synchronization on Linux nodes.
CustomDNSIP:53 (if using custom DNS servers) UDP 53 If you're using custom DNS servers, you must ensure they're accessible by the cluster nodes.
APIServerPublicIP:443 (if running pods/deployments that access the API Server) TCP 443 Required if running pods/deployments that access the API Server, those pods/deployments would use the API IP.

Azure US Government required FQDN / application rules

Destination FQDN Port Use
*.hcp.<location>.cx.aks.containerservice.azure.us HTTPS:443 Required for Node <-> API server communication. Replace <location> with the region where your AKS cluster is deployed.
mcr.microsoft.com HTTPS:443 Required to access images in Microsoft Container Registry (MCR). This registry contains first-party images/charts (for example, coreDNS, etc.). These images are required for the correct creation and functioning of the cluster, including scale and upgrade operations.
*.data.mcr.microsoft.com HTTPS:443 Required for MCR storage backed by the Azure content delivery network (CDN).
management.usgovcloudapi.net HTTPS:443 Required for Kubernetes operations against the Azure API.
login.microsoftonline.us HTTPS:443 Required for Microsoft Entra authentication.
packages.microsoft.com HTTPS:443 This address is the Microsoft packages repository used for cached apt-get operations. Example packages include Moby, PowerShell, and Azure CLI.
acs-mirror.azureedge.net HTTPS:443 This address is for the repository required to install required binaries like kubenet and Azure CNI.

The following FQDN / application rules aren't required, but are recommended for AKS clusters:

Destination FQDN Port Use
security.ubuntu.com, azure.archive.ubuntu.com, changelogs.ubuntu.com HTTP:80 This address lets the Linux cluster nodes download the required security patches and updates.
snapshot.ubuntu.com HTTPS:443 This address lets the Linux cluster nodes download the required security patches and updates from ubuntu snapshot service.

If you choose to block/not allow these FQDNs, the nodes will only receive OS updates when you do a node image upgrade or cluster upgrade. Keep in mind that node image upgrades also come with updated packages including security fixes.

GPU enabled AKS clusters required FQDN / application rules

Destination FQDN Port Use
nvidia.github.io HTTPS:443 This address is used for correct driver installation and operation on GPU-based nodes.
us.download.nvidia.com HTTPS:443 This address is used for correct driver installation and operation on GPU-based nodes.
download.docker.com HTTPS:443 This address is used for correct driver installation and operation on GPU-based nodes.

Windows Server based node pools required FQDN / application rules

Destination FQDN Port Use
onegetcdn.azureedge.net, go.microsoft.com HTTPS:443 To install windows-related binaries
*.mp.microsoft.com, www.msftconnecttest.com, ctldl.windowsupdate.com HTTP:80 To install windows-related binaries

If you choose to block/not allow these FQDNs, the nodes will only receive OS updates when you do a node image upgrade or cluster upgrade. Keep in mind that Node Image Upgrades also come with updated packages including security fixes.

AKS addons and integrations

Microsoft Defender for Containers

Required FQDN / application rules

FQDN Port Use
login.microsoftonline.com
login.microsoftonline.us (Azure Government)
login.microsoftonline.cn (Azure operated by 21Vianet)
HTTPS:443 Required for Active Directory Authentication.
*.ods.opinsights.azure.com
*.ods.opinsights.azure.us (Azure Government)
*.ods.opinsights.azure.cn (Azure operated by 21Vianet)
HTTPS:443 Required for Microsoft Defender to upload security events to the cloud.
*.oms.opinsights.azure.com
*.oms.opinsights.azure.us (Azure Government)
*.oms.opinsights.azure.cn (Azure operated by 21Vianet)
HTTPS:443 Required to Authenticate with LogAnalytics workspaces.

CSI Secret Store

Required FQDN / application rules

FQDN Port Use
vault.azure.net HTTPS:443 Required for CSI Secret Store addon pods to talk to Azure KeyVault server.
*.vault.usgovcloudapi.net HTTPS:443 Required for CSI Secret Store addon pods to talk to Azure KeyVault server in Azure Government.

Azure Monitor for containers

There are two options to provide access to Azure Monitor for containers:

  • Allow the Azure Monitor ServiceTag.
  • Provide access to the required FQDN/application rules.

Required network rules

Destination Endpoint Protocol Port Use
ServiceTag - AzureMonitor:443 TCP 443 This endpoint is used to send metrics data and logs to Azure Monitor and Log Analytics.

Required FQDN / application rules

FQDN Port Use
dc.services.visualstudio.com HTTPS:443 This endpoint is used by Azure Monitor for Containers Agent Telemetry.
*.ods.opinsights.azure.com HTTPS:443 This endpoint is used by Azure Monitor for ingesting log analytics data.
*.oms.opinsights.azure.com HTTPS:443 This endpoint is used by omsagent, which is used to authenticate the log analytics service.
*.monitoring.azure.com HTTPS:443 This endpoint is used to send metrics data to Azure Monitor.
<cluster-region-name>.ingest.monitor.azure.com HTTPS:443 This endpoint is used by Azure Monitor managed service for Prometheus metrics ingestion.
<cluster-region-name>.handler.control.monitor.azure.com HTTPS:443 This endpoint is used to fetch data collection rules for a specific cluster.

Microsoft Azure operated by 21Vianet required FQDN / application rules

FQDN Port Use
dc.services.visualstudio.cn HTTPS:443 This endpoint is used by Azure Monitor for Containers Agent Telemetry.
*.ods.opinsights.azure.cn HTTPS:443 This endpoint is used by Azure Monitor for ingesting log analytics data.
*.oms.opinsights.azure.cn HTTPS:443 This endpoint is used by omsagent, which is used to authenticate the log analytics service.
global.handler.control.monitor.azure.cn HTTPS:443 This endpoint is used by Azure Monitor for accessing the control service.
<cluster-region-name>.handler.control.monitor.azure.cn HTTPS:443 This endpoint is used to fetch data collection rules for a specific cluster.

Azure US Government required FQDN / application rules

FQDN Port Use
dc.services.visualstudio.us HTTPS:443 This endpoint is used by Azure Monitor for Containers Agent Telemetry.
*.ods.opinsights.azure.us HTTPS:443 This endpoint is used by Azure Monitor for ingesting log analytics data.
*.oms.opinsights.azure.us HTTPS:443 This endpoint is used by omsagent, which is used to authenticate the log analytics service.
global.handler.control.monitor.azure.us HTTPS:443 This endpoint is used by Azure Monitor for accessing the control service.
<cluster-region-name>.handler.control.monitor.azure.us HTTPS:443 This endpoint is used to fetch data collection rules for a specific cluster.

Azure Policy

Required FQDN / application rules

FQDN Port Use
data.policy.core.windows.net HTTPS:443 This address is used to pull the Kubernetes policies and to report cluster compliance status to policy service.
store.policy.core.windows.net HTTPS:443 This address is used to pull the Gatekeeper artifacts of built-in policies.
dc.services.visualstudio.com HTTPS:443 Azure Policy add-on that sends telemetry data to applications insights endpoint.

Microsoft Azure operated by 21Vianet required FQDN / application rules

FQDN Port Use
data.policy.azure.cn HTTPS:443 This address is used to pull the Kubernetes policies and to report cluster compliance status to policy service.
store.policy.azure.cn HTTPS:443 This address is used to pull the Gatekeeper artifacts of built-in policies.

Azure US Government required FQDN / application rules

FQDN Port Use
data.policy.azure.us HTTPS:443 This address is used to pull the Kubernetes policies and to report cluster compliance status to policy service.
store.policy.azure.us HTTPS:443 This address is used to pull the Gatekeeper artifacts of built-in policies.

AKS cost analysis add-on

Required FQDN / application rules

FQDN Port Use
management.azure.com
management.usgovcloudapi.net (Azure Government)
management.chinacloudapi.cn (Azure operated by 21Vianet)
HTTPS:443 Required for Kubernetes operations against the Azure API.
login.microsoftonline.com
login.microsoftonline.us (Azure Government)
login.microsoftonline.cn (Azure operated by 21Vianet)
HTTPS:443 Required for Microsoft Entra ID authentication.

Cluster extensions

Required FQDN / application rules

FQDN Port                Use
<region>.dp.kubernetesconfiguration.azure.com HTTPS:443 This address is used to fetch configuration information from the Cluster Extensions service and report extension status to the service.
mcr.microsoft.com, *.data.mcr.microsoft.com HTTPS:443 This address is required to pull container images for installing cluster extension agents on AKS cluster.
arcmktplaceprod.azurecr.io HTTPS:443 This address is required to pull container images for installing marketplace extensions on AKS cluster.
arcmktplaceprod.centralindia.data.azurecr.io HTTPS:443 This address is for the Central India regional data endpoint and is required to pull container images for installing marketplace extensions on AKS cluster.
arcmktplaceprod.japaneast.data.azurecr.io HTTPS:443 This address is for the East Japan regional data endpoint and is required to pull container images for installing marketplace extensions on AKS cluster.
arcmktplaceprod.westus2.data.azurecr.io HTTPS:443 This address is for the West US2 regional data endpoint and is required to pull container images for installing marketplace extensions on AKS cluster.
arcmktplaceprod.westeurope.data.azurecr.io HTTPS:443 This address is for the West Europe regional data endpoint and is required to pull container images for installing marketplace extensions on AKS cluster.
arcmktplaceprod.eastus.data.azurecr.io HTTPS:443 This address is for the East US regional data endpoint and is required to pull container images for installing marketplace extensions on AKS cluster.
*.ingestion.msftcloudes.com, *.microsoftmetrics.com HTTPS:443 This address is used to send agents metrics data to Azure.
marketplaceapi.microsoft.com HTTPS: 443 This address is used to send custom meter-based usage to the commerce metering API.

Azure US Government required FQDN / application rules

FQDN Port Use
<region>.dp.kubernetesconfiguration.azure.us HTTPS:443 This address is used to fetch configuration information from the Cluster Extensions service and report extension status to the service.
mcr.microsoft.com, *.data.mcr.microsoft.com HTTPS:443 This address is required to pull container images for installing cluster extension agents on AKS cluster.

Note

For any addons that aren't explicitly stated here, the core requirements cover it.

Next steps

In this article, you learned what ports and addresses to allow if you want to restrict egress traffic for the cluster.

If you want to restrict how pods communicate between themselves and East-West traffic restrictions within cluster see Secure traffic between pods using network policies in AKS.