AKS Service deployment done and can´t reach external IPs for specific service

AKS-MON-01 1

I deployed two services with azure AKS and cant reach external ip of a specific service, both services are relying on linux containers. I find strange that service pods can run the docker web service via port fowarding, and also have access to internet when I go inside do a curl request. Some nodes access inside the pods arent reacheable, but one pod can reach other pod in the same service.

prmanhas-MSFT 17,886 Reputation points Microsoft Employee

2021-02-26T07:22:09.95+00:00

@AKS-MON-01 Any updates on the issue?

Please "Accept as Answer" if below helped so it can help others in community looking for help on similar topics.

Thanks

11 answers

AKS-MON-01 1

service deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    environment: production
    app: my-app
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: my-app
    spec:
      nodeSelector:
          "beta.kubernetes.io/os": linux
      containers:
      - env:
         - name: PUBLIC_FOLDER
           value: ../site
         - name: HTTP_PORT
           value: "80"
        image: <registry_url>/<registry_user>/<name_image>
        imagePullPolicy: Always
        name: my-app
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            port: 80
            path: /api/
        livenessProbe:
          httpGet:
            port: 80
            path: /api/
      imagePullSecrets:
        - name: regcred
  selector:
    matchLabels:
      app: my-app
---
apiVersion: v1
kind: Service
metadata:
  name: my-app
spec:
  type: "LoadBalancer"
  externalTrafficPolicy: Local
  ports:
  - name: "http"
    protocol: TCP
    port: 80
    targetPort: 80
  selector:
    name: my-app

Service YAML

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"my-app","namespace":"default"},"spec":{"externalTrafficPolicy":"Local","ports":[{"name":"http","port":80,"protocol":"TCP","targetPort":80}],"selector":{"name":"my-app"},"type":"LoadBalancer"}}
  creationTimestamp: "2021-02-19T20:00:00Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
      f:spec:
        f:externalTrafficPolicy: {}
        f:ports:
          .: {}
          k:{"port":80,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:name: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-02-19T20:00:00Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .: {}
          v:"service.kubernetes.io/load-balancer-cleanup": {}
      f:status:
        f:loadBalancer:
          f:ingress: {}
    manager: kube-controller-manager
    operation: Update
    time: "2021-02-19T20:00:15Z"
  name: my-app
  namespace: default
  resourceVersion: "3781154"
  selfLink: /api/v1/namespaces/default/services/my-app
  uid: f5261e77-61dc-4fa3-822d-ed178aef0851
spec:
  clusterIP: 10.0.246.3
  externalTrafficPolicy: Local
  healthCheckNodePort: 31524
  ports:
  - name: http
    nodePort: 31635
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    name: my-app
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 52.149.58.105

Service wide

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
my-app LoadBalancer 10.0.246.3 52.149.58.105 80:31635/TCP 3m30s name=my-app

Service Describe

Name:                     my-app
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 name=my-app
Type:                     LoadBalancer
IP:                       10.0.246.3
LoadBalancer Ingress:     52.149.58.105
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  31635/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     31524
Events:
  Type    Reason                Age    From                Message
  ----    ------                ----   ----                -------
  Normal  EnsuringLoadBalancer  3m38s  service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   3m23s  service-controller  Ensured load balancer

Working Service Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-vote-back
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-vote-back
  template:
    metadata:
      labels:
        app: azure-vote-back
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": linux
      containers:
      - name: azure-vote-back
        image: mcr.microsoft.com/oss/bitnami/redis:6.0.8
        env:
        - name: ALLOW_EMPTY_PASSWORD
          value: "yes"
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 6379
          name: redis
---
apiVersion: v1
kind: Service
metadata:
  name: azure-vote-back
spec:
  ports:
  - port: 6379
  selector:
    app: azure-vote-back
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-vote-front
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-vote-front
  template:
    metadata:
      labels:
        app: azure-vote-front
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": linux
      containers:
      - name: azure-vote-front
        image: mcr.microsoft.com/azuredocs/azure-vote-front:v1
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 80
        env:
        - name: REDIS
          value: "azure-vote-back"
---
apiVersion: v1
kind: Service
metadata:
  name: azure-vote-front
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: azure-vote-front

Working Services yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"azure-vote-front","namespace":"default"},"spec":{"externalIPs":["20.190.19.155"],"ports":[{"port":80}],"selector":{"app":"azure-vote-front"},"type":"LoadBalancer"}}
  creationTimestamp: "2021-02-09T17:36:22Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .: {}
          v:"service.kubernetes.io/load-balancer-cleanup": {}
      f:status:
        f:loadBalancer:
          f:ingress: {}
    manager: kube-controller-manager
    operation: Update
    time: "2021-02-09T17:36:27Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
      f:spec:
        f:externalIPs: {}
        f:externalTrafficPolicy: {}
        f:ports:
          .: {}
          k:{"port":80,"protocol":"TCP"}:
            .: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:app: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-02-16T18:34:59Z"
  name: azure-vote-front
  namespace: default
  resourceVersion: "3151634"
  selfLink: /api/v1/namespaces/default/services/azure-vote-front
  uid: eeebb173-49a1-4d41-94a5-a4058e30f4f4
spec:
  clusterIP: 10.0.50.241
  externalIPs:
  - 20.190.19.155
  externalTrafficPolicy: Cluster
  ports:
  - nodePort: 31317
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: azure-vote-front
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 20.190.19.152

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"azure-vote-back","namespace":"default"},"spec":{"ports":[{"port":6379}],"selector":{"app":"azure-vote-back"}}}
  creationTimestamp: "2021-02-09T17:36:21Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
      f:spec:
        f:ports:
          .: {}
          k:{"port":6379,"protocol":"TCP"}:
            .: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:app: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-02-09T17:36:21Z"
  name: azure-vote-back
  namespace: default
  resourceVersion: "1697371"
  selfLink: /api/v1/namespaces/default/services/azure-vote-back
  uid: ab864d8d-a0b8-4b85-82ae-fe3007f83e2c
spec:
  clusterIP: 10.0.85.65
  ports:
  - port: 6379
    protocol: TCP
    targetPort: 6379
  selector:
    app: azure-vote-back
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Working Service Wide

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
azure-vote-front LoadBalancer 10.0.50.241 20.190.19.152,20.190.19.155 80:31317/TCP 10d app=azure-vote-front

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
azure-vote-back ClusterIP 10.0.85.65 <none> 6379/TCP 10d app=azure-vote-back

Working Service Describe

Name:                     azure-vote-front
Namespace:                default
Labels:                   <none>
Annotations:              <none>
Selector:                 app=azure-vote-front
Type:                     LoadBalancer
IP:                       10.0.50.241
External IPs:             20.190.19.155
LoadBalancer Ingress:     20.190.19.152
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  31317/TCP
Endpoints:                10.244.1.9:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Name:              azure-vote-back
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=azure-vote-back
Type:              ClusterIP
IP:                10.0.85.65
Port:              <unset>  6379/TCP
TargetPort:        6379/TCP
Endpoints:         10.244.1.8:6379
Session Affinity:  None
Events:            <none>

Lucas Camargo Reis 86 Reputation points

2021-02-22T13:27:36.897+00:00
hi @AKS-MON-01

It's look like a problem to azure create a Load Balancer to your service, look for the events in the service my-app.

Normally it's a permission problem between your AKS Cluster and Resource Group that host all AKS Resources.

Please run thi command:

kubectl get events my-app
Please sign in to rate this answer.

0 comments No comments
Sign in to comment
AKS-MON-01 1 Reputation point

2021-02-22T17:55:32.18+00:00
I receive this message:

PS C:\Windows\system32> kubectl get events my-app Error from server (NotFound): events "my-app" not found
Please sign in to rate this answer.

0 comments No comments
Sign in to comment
AKS-MON-01 1 Reputation point

2021-02-23T15:35:54.703+00:00

Let´s assume I have another cluster and this cluster has one load balancer in it.
this new cluster has azure CNI Network .
What do I need to do for my apps to be deployed and accessed with a minimum effort(no ingress, no nginx controllers) like the vote-app?
Please sign in to rate this answer.

0 comments No comments
Sign in to comment
Lucas Camargo Reis 86 Reputation points

2021-02-23T18:44:54.077+00:00

Hi @AKS-MON-01

You need a Service with LoadBalancer type to expose your app/pod or deployment outside your cluster and your Kubernetes cluster [Service Principal or system assigned managed identity] need to have a permission with network contributor inside a Resource Group and VNET that hosted your AKS cluster.

See more informations on this links:
https://learn.microsoft.com/en-us/azure/aks/kubernetes-service-principal#delegate-access-to-other-azure-resources
https://learn.microsoft.com/en-us/azure/aks/internal-lb
Please sign in to rate this answer.

0 comments No comments
Sign in to comment