Health probes are failing after enabling network policy in AKS

Question

Health probes are failing after enabling network policy in AKS

Tanul 1,291

This network policy is causing connection refuse in health probes.. This error is coming

"[http://XX.XX.XX.XX:8080/health/ready]": dial tcp XX.XX.XX.XX:8080: connect: connection refused

Can someone help please.

kind: NetworkPolicy

apiVersion: networking.k8s.io/v1

metadata:

  name: deny-all.policy

  namespace: default 

spec:

  podSelector: {}

  policyTypes:

    - Ingress

    - Egress  

  

---

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: allow.policy

  namespace: default

spec:

  podSelector: {}

  policyTypes:

  - Ingress

  - Egress

  ingress: 

  - from:

    - namespaceSelector:

        matchLabels:

          kubernetes.io/metadata.name: default 

  - from:

    - namespaceSelector:

        matchLabels:

          kubernetes.io/metadata.name: ingress-ns 

    - podSelector:

        matchLabels:

          app.kubernetes.io/name: "test"  

  egress:

  - to:

    - namespaceSelector:

        matchLabels:

          kubernetes.io/metadata.name: default

  - to:

    - namespaceSelector:

        matchLabels:

          kubernetes.io/metadata.name: kube-system

    - podSelector:

        matchLabels:

          k8s-app: kube-dns      

    ports:

    - port: 53

      protocol: UDP

    - port: 53

      protocol: TCP    

    - protocol: TCP

      port: 443

      endPort: 443

Tanul 1,291 Reputation points

2023-07-04T16:21:52.4366667+00:00

I have tried adding host as localhost as well in health probes Still its failing.
KarishmaTiwari-MSFT 20,777 Reputation points Microsoft Employee Moderator

2023-07-18T03:22:48.7066667+00:00

@Tanul Checking in to see if the answer below helped. Eddie has responded to your last comment as well. Please check those and let us know.

If you still have questions, please let us know in the "comments" and we would be happy to help you. Comment is the fastest way of notifying the experts.

If the answer below has been helpful, we appreciate hearing from you and would love to help others who may have the same question. Accepting answers helps increase visibility of this question for other members of the Microsoft Q&A community.

Thank you for helping to improve Microsoft Q&A!

1 answer

Your answer

Tanul 1,291 Reputation points

2023-07-04T16:21:52.4366667+00:00

I have tried adding host as localhost as well in health probes Still its failing.
KarishmaTiwari-MSFT 20,777 Reputation points Microsoft Employee Moderator

2023-07-18T03:22:48.7066667+00:00

@Tanul Checking in to see if the answer below helped. Eddie has responded to your last comment as well. Please check those and let us know.

If you still have questions, please let us know in the "comments" and we would be happy to help you. Comment is the fastest way of notifying the experts.

If the answer below has been helpful, we appreciate hearing from you and would love to help others who may have the same question. Accepting answers helps increase visibility of this question for other members of the Microsoft Q&A community.

Thank you for helping to improve Microsoft Q&A!

Answer 1

Eddie Neto 1,251 Microsoft Employee

Hi @Tanul

Thanks for reaching Microsoft Q&A

Can you confirm once you created your AKS cluster you have enabled any network policy? If yes, which one was it?
If you did not enable any network policy during the creation of the cluster, the network policy that you are trying to apply above will not work.

Look forward to hearing from you.

Tanul 1,291 Reputation points

2023-07-04T18:19:15.1933333+00:00

Its azure.. And its enabled. Yesterday I have deleted and recreated the infra to enable it. I also have azure-npm pods in kube-system namespace
Tanul 1,291 Reputation points

2023-07-05T08:02:06.6866667+00:00

@Eddie Neto Could you please suggest.. None of the health probes are working
Eddie Neto 1,251 Reputation points Microsoft Employee

2023-07-05T08:04:22.25+00:00

@Tanul

Allow me sometime please to test it. I will return to you soon.
Tanul 1,291 Reputation points

2023-07-05T08:22:27.65+00:00

@Eddie Neto Sure please.. My whole team is unable to work because of this.. Kindly try to resolve it as early as possible. I would be grateful
Eddie Neto 1,251 Reputation points Microsoft Employee

2023-07-05T08:25:45.81+00:00

@Tanul

Which AKS cluster version are you running? Can you share the YAML application that you are testing the traffic, please?

Tanul 1,291

@Eddie Neto Its AKS 1.25.6.. I can't share everything.. But this is the main thing which can help u understand.

spec:
      securityContext:
        fsGroup: 9999
        runAsGroup: 9999
        runAsNonRoot: true
        runAsUser: 8888
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: bcgimgintegrationsslack

          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
              - NET_RAW
            readOnlyRootFilesystem: false

          resources:
            limits:
              cpu: 30m
              memory: 180Mi
            requests:
              cpu: 10m
              memory: 70Mi
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /health/ready
              host: localhost
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
            failureThreshold: 3
            successThreshold: 2
          livenessProbe:
            httpGet:
              path: /health/live
              host: localhost
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 30
            failureThreshold: 3
            successThreshold: 1

Tanul 1,291 Reputation points

2023-07-05T08:32:18.5233333+00:00

@Eddie Neto It feels some problem with AKS.. I tested these network policies in K3s and everything is working fine.. Its a panic situation here
Eddie Neto 1,251 Reputation points Microsoft Employee

2023-07-05T10:27:59.0566667+00:00
@Tanul

I´ve been testing here from my side....please add this below on your network policy. Let me know the result.

- ipBlock: cidr: "0.0.0.0/0" except: - "10.0.0.0/8"
Tanul 1,291 Reputation points

2023-07-05T10:54:34.47+00:00

@Eddie Neto In which section Ingress or egress
Is there any provision to allow all ports in ingress
Eddie Neto 1,251 Reputation points Microsoft Employee

2023-07-05T11:08:54.6966667+00:00

@Tanul

Egress....sorry missed to inform above.
Tanul 1,291 Reputation points

2023-07-05T13:26:32.4266667+00:00

@Eddie Neto Now here is the thing. I have added this which you have specified.. And it worked. But after this I reverted the network policy and removed your suggested changes and now everything is working. It seems to be some temporary fix happened at azure-npm pod level.. How is it possible that now it is working without any single change
Tanul 1,291 Reputation points

2023-07-05T13:28:24.6433333+00:00

Since 2 days nothing was working because of this and all of a sudden it started working.. Just by adding your change once.. And even after deleting it.. It seems to be surprising..
Tanul 1,291 Reputation points

2023-07-05T14:02:33.4966667+00:00

@Eddie Neto Any suggestion on this please. Really appreciate
Eddie Neto 1,251 Reputation points Microsoft Employee

2023-07-05T14:04:32.77+00:00

Hi @Tanul

Glad to hear that we were able to fix it. But it's interesting even after removing worked, so the NPM pods might be restarted and the traffic got passed.

From my side is working as expected as I did some tests on your scenario. Please if you face any other behavior raise a ticket to our support team so we can check much more deeper.

Hope this helps. Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.
Tanul 1,291 Reputation points

2023-07-05T14:53:24.64+00:00

@Eddie Neto No but NPM pods never restarted
Tanul 1,291 Reputation points

2023-07-05T14:54:02.84+00:00

@Eddie Neto Is there any way to restart them.. I wanted to confirm if it is permanent fix or temporary
Eddie Neto 1,251 Reputation points Microsoft Employee

2023-07-05T14:56:44.8733333+00:00

@Tanul

You can just delete them and will automatic get re-created.

$ kubectl delete pod -n kube-system azure-npm-xxxxx --force
Tanul 1,291 Reputation points

2023-07-05T17:04:01.9533333+00:00

@Eddie Neto Can I go with this

kubectl rollout restart daemonset azure-npm -n kube-system
Tanul 1,291 Reputation points

2023-07-05T17:44:18.98+00:00

@Eddie Neto Can I run like this
Eddie Neto 1,251 Reputation points Microsoft Employee

2023-07-06T07:52:57.4633333+00:00

Hi @Tanul

Yes, you can as well. Deleting Daemon set you´re going to delete the entire NPM pod through all nodes.

Let me know the result.
Eddie Neto 1,251 Reputation points Microsoft Employee

2023-07-07T07:47:51.68+00:00

Hi Tanul

As I didn't hear from you yesterday, just a quick follow-up to see if everything is working fine?

If this has been helpful, please take a moment to accept answers as this helps increase visibility of this question for other members of the Microsoft Q&A community. Thank you for helping to improve Microsoft Q&A!
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Eddie Neto 1,251 Reputation points Microsoft Employee

2023-07-11T11:08:59.3266667+00:00

Hi Tanul

Any update/issue so far?

If not, hope this helps. Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.

Share via

Health probes are failing after enabling network policy in AKS

1 answer

Your answer