AKS: connect to external service on a different aks cluster on private network

Question

Hi,
my goal is to call a service on an aks cluster (aks1) from a pod or a service on a second aks cluster (aks2).
These clusters will be on different regions and should communicate over a private network.

So, after some reading and some video listening, it seemed for me that the best option was to use an externalName service ok AKS2 calling a service defined in a custom private DNS zone (), being these two VNets paired before.

dev-vnet  10.0.0.0/14  
  
dev-aks  
Service CIDR      10.255.0.0/16  
DNS service IP   10.255.0.10  
Docker bridge CIDR 10.254.0.1/16  
Network policy Azure  
  
=======================================  
dev-test1-aks  v1.22.4 - 1 node  
dev-test1-vnet  11.0.0.0/16  
  
dev-test2-vnet  11.1.0.0/16   

After a lot of trials all I can get is connectivity between pods networks and never to reach the service network from the other cluster.

• I don't see any active firewall
• I've peered all three networks: dev-test1-vnet, dev-test2-vnet, dev-vnet (services CIDR)
• I've create a Private DNS zones private.eu.dev where I've put the "ecommerce" A record (10.0.129.155) that should be resolved by the externalName service

dev-test1-aks (EU cluster):

kubectl create deployment eu-ecommerce --image=k8s.gcr.io/echoserver:1.4 --port=8080 --replicas=1  
kubectl expose deployment eu-ecommerce --type=ClusterIP --port=8080 --name=eu-ecommerce  
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml  
kubectl create ingress eu-ecommerce --class=nginx --rule=eu.ecommerce/*=eu-ecommerce:8080 -o yaml --dry-run=client  

This is one of the externalName I've tried on dev-test2-aks:

apiVersion: v1  
kind: Service  
metadata:  
  name: eu-services  
  namespace: default  
spec:  
  type: ExternalName  
  externalName: ecommerce.private.eu.dev  
  ports:  
    - port: 8080  
      protocol: TCP  

These are some of my tests:

# --- Test externalName   
kubectl --context=dev-test2-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://eu-services:8080  
: '  
    wget: cant connect to remote host (10.0.129.155): Connection timed out  
'  
  
# --- Test connectivity AKS1 -> eu-ecommerce service  
kubectl --context=dev-test1-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://eu-ecommerce:8080  
kubectl --context=dev-test1-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://10.0.129.155:8080  
kubectl --context=dev-test1-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://eu-ecommerce.default.svc.cluster.local:8080  
kubectl --context=dev-test1-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://ecommerce.private.eu.dev:8080  
# OK client_address=11.0.0.11  
  
# --- Test connectivity AKS2 -> eu-ecommerce POD  
kubectl --context=dev-test2-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://11.0.0.103:8080  
#> OK  
  
# --- Test connectivity AKS2 -> eu-ecommerce service  
kubectl --context=dev-test2-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://ecommerce.private.eu.dev:8080  
#> FAIL  
kubectl --context=dev-test2-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget -qO- http://10.0.129.155:8080  
  
  
# --- Test connectivity - LB private IP  
kubectl --context=dev-test1-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget --no-cache -qO- http://lb.private.eu.dev/ecommerce  
#> OK  
kubectl --context=dev-test2-aks run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox -- wget --no-cache -qO- http://lb.private.eu.dev/ecommerce  
#> KO  

I've also created inbound and outbound network policies for the aks networks:

• on dev-aks allow all incoming from 11.1/16 and 11.0/16
• on dev-test2-aks allow any outbound

Doing the same network test from the same virtual machine, but in two different point of azure portal give different results. The first is from the virtual machines view (../virtualMachines/network-debug/vmConnectivity), the second from the virtual network overview (/virtualNetworks/dev-x-test2-vnet/connectionTroubleshoot).

EDIT:
I'd like to stress that I can reach the other pod network but not the services

Related docs:

• https://learn.microsoft.com/en-us/azure/aks/private-clusters#virtual-network-peering
• https://kubernetes.io/docs/concepts/services-networking/service/#externalname
• https://learn.microsoft.com/en-us/azure/dns/private-dns-getstarted-portal#create-a-private-dns-zone
• https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview
• https://www.youtube.com/watch?v=J4S6AxYNDtM

Answer 1

@AlessandroAffinito-9728

Welcome to Microsoft Q&A Platform, thanks for posting your query here.

Firstly, apologies for the delay in responding here and any inconvenience this issue may have caused.

Thanks for sharing the issue description and your environment setup in detail.

The issue is mostly with the A record (I've create a Private DNS zones private.eu.dev where I've put the "ecommerce" A record (10.0.129.155) that should be resolved by the externalName service)

Please change this to IP of the LB http://lb.private.eu.dev/ecommerce and this should solve the issue.

Hope that helps.
If the suggested response helped you resolve your issue, please 'Accept as answer', so that it can help others in the community looking for help on similar topics.