How to configure the egress of pods for connecting to other Azure services through their private endpoints?

Question

How to configure the egress of pods for connecting to other Azure services through their private endpoints?

Bergs 60

For reference, our organization's Azure account is being managed by a 3rd-party who specialized on security. I was given a limited access to our AKS clusters that are newly created. I'm not sure how much they changed it from the default configuration. What I found are all AKS clusters using Azure CNI Node Subnet with Calico as Network policy, and some of the clusters are sharing VNET subnets with thousands of IP addresses.

They closed public accesses to most of the other Azure services/resources(Container Registries, Service Busses, Storage Accounts, App Services, etc.) and can only be accessed through their private endpoints, but they also put a firewall around it too.

We're about to try using NAT Gateway as egress, but those handling the firewall says that it won't work because it will access the aforementioned services publicly instead of the private endpoint. What are my choices for the egress?

Also, please include how to back-up the configuration and to revert the changes. Thanks!

Bergs 60 Reputation points

2024-09-19T01:33:55.8966667+00:00

@Prrudram-MSFT

Regarding your question about egress options, you mentioned that your organization is planning to use NAT Gateway as egress. However, your security team has concerns that it will access the services publicly instead of the private endpoint.

Yes, something like that. All of the resources like Service Buses, Container Registries, and Storage Accounts aren't accessible publicly anymore, and can only be accessed through private endpoints. That's how much I can see.

Regarding the Azure Firewalls, the 3rd-party is the one managing it and I can't access it. I'll relay this on them instead, but on the AKS side, what are the things I need to set.

There's a chance that they already considered this but they refuse because of how large the IP addresses they allotted for each subnet. It is their design and I can't do anything about it.

Another option is to contain the ip addresses of the pods created to a smaller range that is within the allotted ip addresses of the VNET's subnet, but how can I be sure that these ip addresses will be reserved to the pods of that AKS cluster and won't be touched by the other services?
Prrudram-MSFT 28,286 Reputation points Microsoft Employee Moderator

2024-09-19T06:15:25.75+00:00

Hello @Bergs

If your organization is concerned about accessing services publicly instead of through private endpoints, you can use Azure Private Link to access Azure services over a private endpoint. Azure Private Link provides private connectivity from a virtual network to Azure services. With Private Link, you can access Azure services such as Storage Accounts, Azure SQL Database, and Azure Cosmos DB over a private endpoint within your virtual network. This way, you can ensure that your traffic stays within your virtual network and does not traverse the public internet. Regarding the AKS side, you can restrict egress traffic from your AKS cluster to only access defined ports and addresses. You can do this by using Azure Firewall or any other outbound restriction method or appliance.

https://learn.microsoft.com/en-us/azure/aks/limit-egress-traffic?tabs=aks-with-system-assigned-identities

It contains the cluster requirements for a base AKS deployment, and additional requirements for optional addons and features. If your security team has already allotted large IP addresses for each subnet, you can still contain the IP addresses of the pods created to a smaller range that is within the allotted IP addresses of the VNET's subnet. You can do this by using Kubernetes Network Policies. With Network Policies, you can allow or deny specific network paths within the cluster based on namespaces and label selectors. This way, you can ensure that the IP addresses reserved for the pods of your AKS cluster won't be touched by other services.

I hope this helps. Let me know if you have any further questions.
Patchfox 4,176 Reputation points

2024-09-19T08:03:44.7933333+00:00

In your case I think you need something like AKS CNI flat design with pod subnets

https://learn.microsoft.com/en-us/azure/aks/concepts-network-cni-overview

https://learn.microsoft.com/en-us/azure/aks/concepts-network-azure-cni-pod-subnet

Are there any other services in the planned aks subnet? Or do you mean the possible IP assignments of private links that could collide with the pod IPs?
Bergs 60 Reputation points

2024-09-19T09:07:46.63+00:00

@Prrudram-MSFT yes, I already mentioned on my original question that we have to connect to the other Azure services through their private endpoints because all of the public accesses are already closed off. The reason why we need an egress for the pods is because the private endpoints have a firewall too(mentioned on my original post), and we're going to whitelist the said egress/gateway to that firewall. Allowing the whole subnet of the VNET is not an option because of the huge range of the IP addresses, it can easily surpass a few thousand of IP addresses. I have no control on the VNETs' subnets and the firewalls, the 3rd-party organization is the one controlling it and I have no say on it either. We have to play with their rules.
Bergs 60 Reputation points

2024-09-19T09:21:03.2866667+00:00

@Patchfox We are currently using the Azure CNI Node Subnet, which I assume already the flat network you're talking about, but I think the Azure CNI Overlay would fit with what we need. The 3rd-party who manages our Azure account closed off public access to the Azure services and can only be accessed through private endpoints, but at the same time, they put a firewall in front of it too. So the plan is to make the pods use an egress/gateway and then whitelist the gateway through the firewalls. I don't think we need to access the pods directly through their ip addresses, but instead through ingresses.

But if we're going through this route, can we assign a static IP address on the NAT mentioned here? Or ensure that it is easier for those who manages the firewalls to let it through? Sorry, I have no idea how the firewalls are being configured, nor I have access on it.

It is quite hard to test things out here, especially I have little access on the resources, and some of the changes cannot be reverted, like migrating from Azure CNI Node Subnet to Azure CNI Overlay. And I have no idea how the whole network was designed and the firewalls. I can't delegate the task to them either, because they have too little knowledge about Kubernetes. My knowledge with Kubernetes is skewed on the developer side and networking is actually something I really struggle on.
Patchfox 4,176 Reputation points

2024-09-19T10:09:12.7166667+00:00
Ok Bergs, I understand it now.
Then I would recommend using one of both options:

Internal Load Balancer (Layer 3/4) in front of the Pods (as backbone resources)

Here you can configure a static frontend IP to whitelist or generally manage it via the 3rd party firewall.

Use User Defined Routing

Define a CIDR for the Pods and communicate it with the teams, especially the firewall team/provider. And forward the public traffic of the CIDR resources to the firewall.

1 answer

Your answer

Bergs 60 Reputation points

2024-09-19T01:33:55.8966667+00:00

@Prrudram-MSFT

Regarding your question about egress options, you mentioned that your organization is planning to use NAT Gateway as egress. However, your security team has concerns that it will access the services publicly instead of the private endpoint.

Yes, something like that. All of the resources like Service Buses, Container Registries, and Storage Accounts aren't accessible publicly anymore, and can only be accessed through private endpoints. That's how much I can see.

Regarding the Azure Firewalls, the 3rd-party is the one managing it and I can't access it. I'll relay this on them instead, but on the AKS side, what are the things I need to set.

There's a chance that they already considered this but they refuse because of how large the IP addresses they allotted for each subnet. It is their design and I can't do anything about it.

Another option is to contain the ip addresses of the pods created to a smaller range that is within the allotted ip addresses of the VNET's subnet, but how can I be sure that these ip addresses will be reserved to the pods of that AKS cluster and won't be touched by the other services?
Prrudram-MSFT 28,286 Reputation points Microsoft Employee Moderator

2024-09-19T06:15:25.75+00:00

Hello @Bergs

If your organization is concerned about accessing services publicly instead of through private endpoints, you can use Azure Private Link to access Azure services over a private endpoint. Azure Private Link provides private connectivity from a virtual network to Azure services. With Private Link, you can access Azure services such as Storage Accounts, Azure SQL Database, and Azure Cosmos DB over a private endpoint within your virtual network. This way, you can ensure that your traffic stays within your virtual network and does not traverse the public internet. Regarding the AKS side, you can restrict egress traffic from your AKS cluster to only access defined ports and addresses. You can do this by using Azure Firewall or any other outbound restriction method or appliance.

https://learn.microsoft.com/en-us/azure/aks/limit-egress-traffic?tabs=aks-with-system-assigned-identities

It contains the cluster requirements for a base AKS deployment, and additional requirements for optional addons and features. If your security team has already allotted large IP addresses for each subnet, you can still contain the IP addresses of the pods created to a smaller range that is within the allotted IP addresses of the VNET's subnet. You can do this by using Kubernetes Network Policies. With Network Policies, you can allow or deny specific network paths within the cluster based on namespaces and label selectors. This way, you can ensure that the IP addresses reserved for the pods of your AKS cluster won't be touched by other services.

I hope this helps. Let me know if you have any further questions.
Patchfox 4,176 Reputation points

2024-09-19T08:03:44.7933333+00:00

In your case I think you need something like AKS CNI flat design with pod subnets

https://learn.microsoft.com/en-us/azure/aks/concepts-network-cni-overview

https://learn.microsoft.com/en-us/azure/aks/concepts-network-azure-cni-pod-subnet

Are there any other services in the planned aks subnet? Or do you mean the possible IP assignments of private links that could collide with the pod IPs?
Bergs 60 Reputation points

2024-09-19T09:07:46.63+00:00

@Prrudram-MSFT yes, I already mentioned on my original question that we have to connect to the other Azure services through their private endpoints because all of the public accesses are already closed off. The reason why we need an egress for the pods is because the private endpoints have a firewall too(mentioned on my original post), and we're going to whitelist the said egress/gateway to that firewall. Allowing the whole subnet of the VNET is not an option because of the huge range of the IP addresses, it can easily surpass a few thousand of IP addresses. I have no control on the VNETs' subnets and the firewalls, the 3rd-party organization is the one controlling it and I have no say on it either. We have to play with their rules.
Bergs 60 Reputation points

2024-09-19T09:21:03.2866667+00:00

@Patchfox We are currently using the Azure CNI Node Subnet, which I assume already the flat network you're talking about, but I think the Azure CNI Overlay would fit with what we need. The 3rd-party who manages our Azure account closed off public access to the Azure services and can only be accessed through private endpoints, but at the same time, they put a firewall in front of it too. So the plan is to make the pods use an egress/gateway and then whitelist the gateway through the firewalls. I don't think we need to access the pods directly through their ip addresses, but instead through ingresses.

But if we're going through this route, can we assign a static IP address on the NAT mentioned here? Or ensure that it is easier for those who manages the firewalls to let it through? Sorry, I have no idea how the firewalls are being configured, nor I have access on it.

It is quite hard to test things out here, especially I have little access on the resources, and some of the changes cannot be reverted, like migrating from Azure CNI Node Subnet to Azure CNI Overlay. And I have no idea how the whole network was designed and the firewalls. I can't delegate the task to them either, because they have too little knowledge about Kubernetes. My knowledge with Kubernetes is skewed on the developer side and networking is actually something I really struggle on.
Patchfox 4,176 Reputation points

2024-09-19T10:09:12.7166667+00:00

Ok Bergs, I understand it now.
Then I would recommend using one of both options:

Internal Load Balancer (Layer 3/4) in front of the Pods (as backbone resources)

Here you can configure a static frontend IP to whitelist or generally manage it via the 3rd party firewall.

Use User Defined Routing

Define a CIDR for the Pods and communicate it with the teams, especially the firewall team/provider. And forward the public traffic of the CIDR resources to the firewall.

Answer 1

Hello @Bergs

If I understand correctly, I see that you said your organization has implemented a secure environment for your AKS clusters. Regarding your question about egress options, you mentioned that your organization is planning to use NAT Gateway as egress. However, your security team has concerns that it will access the services publicly instead of the private endpoint.
In this case, you may want to consider using Azure Firewall to protect your AKS clusters and secure outbound and inbound traffic. Azure Firewall can be used to restrict egress traffic and provide secure access to external resources. You can use Azure Firewall to create a DNAT rule that translates the public IP address of the firewall to the private IP address of the service endpoint. This way, your AKS cluster can access the services through the private endpoint, and the traffic will be secured by Azure Firewall. To back up the configuration of your AKS cluster, you can use the az aks show command to retrieve the current configuration of your cluster. You can also use the az aks get-credentials command to download the Kubernetes configuration file for your cluster.
This file contains the configuration of your cluster, including the API server endpoint, authentication credentials, and cluster certificate. To revert the changes, you can use the az aks update command to update the configuration of your AKS cluster. You can also use the Kubernetes configuration file to revert the changes made to your cluster.

I hope this helps! Let me know if you have any other questions.

Share via

How to configure the egress of pods for connecting to other Azure services through their private endpoints?

1 answer

Your answer