แก้ไข

แชร์ผ่าน


Use an internal load balancer with Azure Kubernetes Service (AKS)

You can create and use an internal load balancer to restrict access to your applications in Azure Kubernetes Service (AKS). An internal load balancer doesn't have a public IP and makes a Kubernetes service accessible only to applications that can reach the private IP. These applications can be within the same VNET or in another VNET through VNET peering. This article shows you how to create and use an internal load balancer with AKS.

Note

Azure Load Balancer is available in two SKUs: Basic and Standard. The Standard SKU is used by default when you create an AKS cluster. When you create a LoadBalancer service type, you'll get the same load balancer type as when you provisioned the cluster. For more information, see Azure Load Balancer SKU comparison.

Before you begin

Create an internal load balancer

  1. Create a service manifest named internal-lb.yaml with the service type LoadBalancer and the azure-load-balancer-internal annotation.

    apiVersion: v1
    kind: Service
    metadata:
      name: internal-app
      annotations:
        service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    spec:
      type: LoadBalancer
      ports:
      - port: 80
      selector:
        app: internal-app
    
  2. Deploy the internal load balancer using the kubectl apply command. This command creates an Azure load balancer in the node resource group connected to the same virtual network as your AKS cluster.

    kubectl apply -f internal-lb.yaml
    
  3. View the service details using the kubectl get service command.

    kubectl get service internal-app
    

    The IP address of the internal load balancer is shown in the EXTERNAL-IP column, as shown in the following example output. In this context, External refers to the external interface of the load balancer. It doesn't mean that it receives a public, external IP address. This IP address is dynamically assigned from the same subnet as the AKS cluster.

    NAME           TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
    internal-app   LoadBalancer   10.0.248.59   10.240.0.7    80:30555/TCP   2m
    

Specify an IP address

When you specify an IP address for the load balancer, the specified IP address must reside in the same virtual network as the AKS cluster, but it can't already be assigned to another resource in the virtual network. For example, you shouldn't use an IP address in the range designated for the Kubernetes subnet within the AKS cluster. Using an IP address that's already assigned to another resource in the same virtual network can cause issues with the load balancer.

You can use the az network vnet subnet list Azure CLI command or the Get-AzVirtualNetworkSubnetConfig PowerShell cmdlet to get the subnets in your virtual network.

For more information on subnets, see Add a node pool with a unique subnet.

If you want to use a specific IP address with the load balancer, you have two options: set service annotations or add the LoadBalancerIP property to the load balancer YAML manifest.

Important

Adding the LoadBalancerIP property to the load balancer YAML manifest is deprecating following upstream Kubernetes. While current usage remains the same and existing services are expected to work without modification, we highly recommend setting service annotations instead.

  1. Set service annotations using service.beta.kubernetes.io/azure-load-balancer-ipv4 for an IPv4 address and service.beta.kubernetes.io/azure-load-balancer-ipv6 for an IPv6 address.

    apiVersion: v1
    kind: Service
    metadata:
      name: internal-app
      annotations:
        service.beta.kubernetes.io/azure-load-balancer-ipv4: 10.240.0.25
        service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    spec:
      type: LoadBalancer
      ports:
      - port: 80
      selector:
        app: internal-app
    
  1. View the service details using the kubectl get service command.

    kubectl get service internal-app
    

    The IP address in the EXTERNAL-IP column should reflect your specified IP address, as shown in the following example output:

    NAME           TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
    internal-app   LoadBalancer   10.0.184.168   10.240.0.25   80:30225/TCP   4m
    

For more information on configuring your load balancer in a different subnet, see Specify a different subnet

Before you begin

  1. Create a service manifest named internal-lb-pls.yaml with the service type LoadBalancer and the azure-load-balancer-internal and azure-pls-create annotations. For more options, refer to the Azure Private Link Service Integration design document.

    apiVersion: v1
    kind: Service
    metadata:
      name: internal-app
      annotations:
        service.beta.kubernetes.io/azure-load-balancer-internal: "true"
        service.beta.kubernetes.io/azure-pls-create: "true"
    spec:
      type: LoadBalancer
      ports:
      - port: 80
      selector:
        app: internal-app
    
  2. Deploy the internal load balancer using the kubectl apply command. This command creates an Azure load balancer in the node resource group connected to the same virtual network as your AKS cluster. It also creates a Private Link Service object that connects to the frontend IP configuration of the load balancer associated with the Kubernetes service.

    kubectl apply -f internal-lb-pls.yaml
    
  3. View the service details using the kubectl get service command.

    kubectl get service internal-app
    

    The IP address of the internal load balancer is shown in the EXTERNAL-IP column, as shown in the following example output. In this context, External refers to the external interface of the load balancer. It doesn't mean that it receives a public, external IP address.

    NAME           TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
    internal-app   LoadBalancer   10.125.17.53  10.125.0.66   80:30430/TCP   64m
    
  4. View the details of the Private Link Service object using the az network private-link-service list command.

    # Create a variable for the node resource group
    
    AKS_MC_RG=$(az aks show -g myResourceGroup --name myAKSCluster --query nodeResourceGroup -o tsv)
    
    # View the details of the Private Link Service object
    
    az network private-link-service list -g $AKS_MC_RG --query "[].{Name:name,Alias:alias}" -o table
    

    Your output should look similar to the following example output:

    Name      Alias
    --------  -------------------------------------------------------------------------
    pls-xyz   pls-xyz.abc123-defg-4hij-56kl-789mnop.eastus2.azure.privatelinkservice
    

A Private Endpoint allows you to privately connect to your Kubernetes service object via the Private Link Service you created.

  • Create the private endpoint using the az network private-endpoint create command.

    # Create a variable for the private link service
    
    AKS_PLS_ID=$(az network private-link-service list -g $AKS_MC_RG --query "[].id" -o tsv)
    
    # Create the private endpoint
    
    $ az network private-endpoint create \
        -g myOtherResourceGroup \
        --name myAKSServicePE \
        --vnet-name myOtherVNET \
        --subnet pe-subnet \
        --private-connection-resource-id $AKS_PLS_ID \
        --connection-name connectToMyK8sService
    

PLS Customizations via Annotations

The following are annotations that can be used to customize the PLS resource.

Annotation Value Description Required Default
service.beta.kubernetes.io/azure-pls-create "true" Boolean indicating whether a PLS needs to be created. Required
service.beta.kubernetes.io/azure-pls-name <PLS name> String specifying the name of the PLS resource to be created. Optional "pls-<LB frontend config name>"
service.beta.kubernetes.io/azure-pls-resource-group Resource Group name String specifying the name of the Resource Group where the PLS resource will be created Optional MC_ resource
service.beta.kubernetes.io/azure-pls-ip-configuration-subnet <Subnet name> String indicating the subnet to which the PLS will be deployed. This subnet must exist in the same VNET as the backend pool. PLS NAT IPs are allocated within this subnet. Optional If service.beta.kubernetes.io/azure-load-balancer-internal-subnet, this ILB subnet is used. Otherwise, the default subnet from config file is used.
service.beta.kubernetes.io/azure-pls-ip-configuration-ip-address-count [1-8] Total number of private NAT IPs to allocate. Optional 1
service.beta.kubernetes.io/azure-pls-ip-configuration-ip-address "10.0.0.7 ... 10.0.0.10" A space separated list of static IPv4 IPs to be allocated. (IPv6 is not supported right now.) Total number of IPs should not be greater than the ip count specified in service.beta.kubernetes.io/azure-pls-ip-configuration-ip-address-count. If there are fewer IPs specified, the rest are dynamically allocated. The first IP in the list is set as Primary. Optional All IPs are dynamically allocated.
service.beta.kubernetes.io/azure-pls-fqdns "fqdn1 fqdn2" A space separated list of fqdns associated with the PLS. Optional []
service.beta.kubernetes.io/azure-pls-proxy-protocol "true" or "false" Boolean indicating whether the TCP PROXY protocol should be enabled on the PLS to pass through connection information, including the link ID and source IP address. Note that the backend service MUST support the PROXY protocol or the connections will fail. Optional false
service.beta.kubernetes.io/azure-pls-visibility "sub1 sub2 sub3 … subN" or "*" A space separated list of Azure subscription ids for which the private link service is visible. Use "*" to expose the PLS to all subs (Least restrictive). Optional Empty list [] indicating role-based access control only: This private link service will only be available to individuals with role-based access control permissions within your directory. (Most restrictive)
service.beta.kubernetes.io/azure-pls-auto-approval "sub1 sub2 sub3 … subN" A space separated list of Azure subscription ids. This allows PE connection requests from the subscriptions listed to the PLS to be automatically approved. This only works when visibility is set to "*". Optional []

Use private networks

When you create your AKS cluster, you can specify advanced networking settings. These settings allow you to deploy the cluster into an existing Azure virtual network and subnets. For example, you can deploy your AKS cluster into a private network connected to your on-premises environment and run services that are only accessible internally.

For more information, see configure your own virtual network subnets with Kubenet or with Azure CNI.

You don't need to make any changes to the previous steps to deploy an internal load balancer that uses a private network in an AKS cluster. The load balancer is created in the same resource group as your AKS cluster, but it's instead connected to your private virtual network and subnet, as shown in the following example:

$ kubectl get service internal-app

NAME           TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
internal-app   LoadBalancer   10.1.15.188   10.0.0.35     80:31669/TCP   1m

Note

The cluster identity used by the AKS cluster must at least have the Network Contributor role on the virtual network resource. You can view the cluster identity using the az aks show command, such as az aks show --resource-group <resource-group-name> --name <cluster-name> --query "identity". You can assign the Network Contributor role using the az role assignment create command, such as az role assignment create --assignee <identity-resource-id> --scope <virtual-network-resource-id> --role "Network Contributor".

If you want to define a custom role instead, you need the following permissions:

  • Microsoft.Network/virtualNetworks/subnets/join/action
  • Microsoft.Network/virtualNetworks/subnets/read

For more information, see Add, change, or delete a virtual network subnet.

Specify a different subnet

  • Add the azure-load-balancer-internal-subnet annotation to your service to specify a subnet for your load balancer. The subnet specified must be in the same virtual network as your AKS cluster. When deployed, the load balancer EXTERNAL-IP address is part of the specified subnet.

    apiVersion: v1
    kind: Service
    metadata:
      name: internal-app
      annotations:
        service.beta.kubernetes.io/azure-load-balancer-internal: "true"
        service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "apps-subnet"
    spec:
      type: LoadBalancer
      ports:
      - port: 80
      selector:
        app: internal-app
    

Delete the load balancer

The load balancer is deleted when all of its services are deleted.

As with any Kubernetes resource, you can directly delete a service, such as kubectl delete service internal-app, which also deletes the underlying Azure load balancer.

Next steps

To learn more about Kubernetes services, see the Kubernetes services documentation.