Share via

Back-off restarting failed container in AKS Cluster

MOHAMMED RASHID 26 Reputation points
2022-02-28T19:51:15.29+00:00

I have attach 2 managed disk to AKS Cluster. Attach successfully but pods got fail of both services Postgres and elastiscearch.

The Managed Disk i have same region and location and zone in both disk and aks cluster

Here is the yaml file of elasticsearch

apiVersion: apps/v1 kind: Deployment metadata: name: postgres spec: replicas: 1 selector: matchLabels: app: postgres template: metadata: labels: app: postgres
spec: containers: - name: postgres image: postgres:10.2 ports: - containerPort: xxxx command: - docker-entrypoint.sh - -c - 'shared_buffers=256MB' - -c - 'max_connections=300' envFrom: - configMapRef: name: postgres-config volumeMounts: - mountPath: "/var/lib/postgresql/data" name: postgres-pv-storageclaim volumes: - name: postgres-pv-storageclaim azureDisk: kind: Managed diskName: pgdata diskURI: /subscriptions/SUBSCRIPTION-ID/resourceGroups/MC_devOps_West US 2/providers/Microsoft.Compute/disks/pgdata

Here is yaml of elasticsearch

apiVersion: apps/v1 kind: Deployment metadata: name: elasticsearch spec: replicas: 1 selector: matchLabels: app: elasticsearch template: metadata: labels: app: elasticsearch spec: initContainers: - name: init-sysctl image: busybox:1.27.2 command: - sysctl - -w - vm.max_map_count=262144 securityContext: privileged: true containers: - name: elasticsearch image: elasticsearch:6.8.12 ports: - containerPort: xxx envFrom: - configMapRef: name: elastic-config volumeMounts: - mountPath: "/usr/share/elasticsearch/data" name: elasticsearch-pv-storageclaim volumes: - name: elasticsearch-pv-storageclaim azureDisk: kind: Managed diskName: esdata diskURI: /subscriptions/SUBSCRIPTION-ID/resourceGroups/MC_devOps_West US 2/providers/Microsoft.Compute/disks/esdata

Both yaml so many time i have used it only i add azureDisk in both except that everything is working fine.

I dont know why pods are getting failed

Here is the kubectl describe of both postgres and elastiscearch

Name: postgres-7664569b7d-4tnmq Namespace: default Priority: 0 Node: aks-asecpool-10214-vmss00014/10.240.0.6 Start Time: Mon, 28 Feb 2022 21:52:14 +0530 Labels: app=postgres pod-template-hash=76645b7d Status: Running IP: 10.244.x.x IPs: IP: 10.244.x.x Controlled By: ReplicaSet/postgres-7664569b7d Containers: fx-postgres: Container ID: containerd://1ae785ed6d8d88a4493e322c48f76fb95fcdd13699 Image: postgres:10.2 Image ID: docker.io/library/postgres@sha256:7361bae1fbf5642099663d1f00d0a8806640a617 Port: xxxx/TCP Host Port: 0/TCP Command: docker-entrypoint.sh -c shared_buffers=256MB -c max_connections=300 State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Mon, 28 Feb 2022 21:53:27 +0530 Finished: Mon, 28 Feb 2022 21:53:27 +0530 Ready: False Restart Count: 3 Environment Variables from: postgres-config ConfigMap Optional: false Environment: <none> Mounts: /var/lib/postgresql/data from postgres-pv-storageclaim (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-ft-access-585fb (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: postgres-pv-storageclaim: Type: AzureDisk (an Azure Data Disk mount on the host and bind mount to the pod) DiskName: pgdata DiskURI: /subscriptions/SUBSCRIPTION-ID/resourceGroups/MC_devOps_West US 2/providers/Microsoft.Compute/disks/pgdata Kind: Managed FSType: ext4 CachingMode: ReadWrite ReadOnly: false kube-ft-access-585fb: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 2m1s default-scheduler Successfully assigned default/postgres-7664569b7d-4tnmq to aks-asecpool-10214-vmss00014 Normal SuccessfulAttachVolume 106s attachdetach-controller AttachVolume.Attach succeeded for volume "/subscriptions/SUBSCRIPTION-ID/resourceGroups/MC_devOps_West US 2/providers/Microsoft.Compute/disks/pgdata" Normal Pulled 49s (x4 over 89s) kubelet Container image "postgres:10.2" already present on machine Normal Created 49s (x4 over 89s) kubelet Created container postgres Normal Started 49s (x4 over 89s) kubelet Started container postgres Warning BackOff 12s (x8 over 88s) kubelet Back-off restarting failed container

Azure Container Registry
Azure Container Registry
An Azure service that provides a registry of Docker and Open Container Initiative images.
428 questions
Azure Container Instances
Azure Container Instances
An Azure service that provides customers with a serverless container experience.
676 questions
Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
1,999 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. shiva patpi 13,171 Reputation points Microsoft Employee
    2022-02-28T20:49:20.04+00:00

    Hello @MOHAMMED RASHID ,
    From the logs it is going to Crashloopback state with exit Code : 1

    Reason: CrashLoopBackOff
    Last State: Terminated
    Reason: Error
    Exit Code: 1

    Can you validate the command line arguments: (Just check if that script is accessible, Check if that script is running - try running it manually? , Try to remove that command and create the POD ?
    Command:
    docker-entrypoint.sh

    -
    cshared_buffers=256MB

    -
    cmax_connections=300

    Also validate the code of docker-entrypoint.sh