How to get list of failed requests on Ingress application gateway in AKS

sai theja 0

There are continues alert of failed request on ingress app gateway, but unable to track down to the resource causing for failed requests.

These alerts are from AKS app gateway.

Help me on it.

Backend health of app gateway is verified and all are active, health probe have active status to all backend poo available nodes.

vipullag-MSFT 24,206 Reputation points Microsoft Employee

2023-02-02T07:43:59.6266667+00:00

Hello sai theja

Firstly, apologies for the delay in responding here and any inconvenience this issue may have caused.

This issue needs deeper investigation. Support team will be able to check and help on this. I would recommend you to open a azure support case.

If you don't have a support plan enabled on your subscription, I request you to send an email to AzCommunity@Microsoft.com with Subject as "Attn:Vikas" referencing this thread along with your subscription ID. I can then enable your subscription with a one-time free technical support.

Once the issue is resolved, request you to post the resolution here for the benefit of community.
sai theja 0 Reputation points

2023-03-26T05:22:09.0966667+00:00

Hi @vipullag-MSFT ,

Thanks for reply, vipul.

Apology on late update.

We have enabled the WAF, and diagnostic setting on ingress.

In diagnostic setting containers are configured for the content of logs and which logs to be captured.

From the logs, we mapped the RuleID which are in prevention mode from WAF.

From there we acquired the logs and failure requests.

Thanks,

D.sai theja

2 answers

Adrian Dobrescu 261 Reputation points Microsoft Employee

2023-01-20T15:41:05.5366667+00:00

Hello @sai theja
-You can check the logs of AGIC pod via kubectl logs command:
#kubectl logs -l appgw-ingress=true -n <namespace-name> --tail=100

//This command will retrieve the last 100 lines of logs for the AGIC pod, and will filter the pod based on the label appgw-ingress=true.

-You can also use the --since flag to get logs since specific time or duration.
kubectl logs -l appgw-ingress=true -n ingress-basic --since=5m // it will retrieve you the logs for the past 5 minutes
-Real time logs with -f flag:

**kubectl logs -f -l appgw-ingress=true -n ingress-basic
**
Please "Accept as Answer" and Upvote if it helped, so that it can help others in the community looking for help on similar topics.

Thank you!
Please sign in to rate this answer.
Adrian Dobrescu 261 Reputation points Microsoft Employee

2023-01-20T15:42:26.72+00:00

Once you retrieved the logs, you can use grep or other command-line tools to search for specific keywords or patterns that indicate a failed request, such as "404" or "500".

It's important to note that you need to have the proper access to the cluster and the logs, also make sure you fill the proper namespace where AGIC is running and with the correct label

sai theja 0 Reputation points

2023-01-21T11:30:48.7433333+00:00

Hi @Adrian Dobrescu ,

I have tried with provided commands but not succeeded.

I have used below commands.

kubectl logs <pod-name> <container-name>=true -n <namespace>.

But no result, Output shown as no resource available as

the container is not matching as per my assumption.

I got container name from below command:

kubectl get pods <pod-name> -o custom-columns=name:.spec.containers[*].name -n <namespace>

Where i tried with

kubectl get pods <pod-name> -o custom-columns=name:.spec.containers.name -n <namespace>

No result with this.

Help me on this.
Sign in to comment
Sai Theja 21 Reputation points

2023-01-21T07:25:05.34+00:00
Hi @Adrian Dobrescu ,

Thanks for the reply, But i am unable to retrieve logs.

I tried with the below command.

where container name i received as none,

When i executed:

1.kubectl get pods <pod name> -o custom-columns=name:.spec.containers.name -n <name-space>

But when i executed:

kubectl get pods <pod name> -o custom-columns=name:.spec.containers[*].name -n <name-space>

received container name as "ingress-azure".

Tried with this container name:

Kubectl get pods <pod-name> <container-name> -n <name-space> in this command but i received as not valid for the pod because i took from overall spec, but there is no container i received as per my first command.

Kindly help me on this.

Thanks.
Please sign in to rate this answer.
Adrian Dobrescu 261 Reputation points Microsoft Employee

2023-01-21T10:13:52.9133333+00:00

Hello @sai theja ,

You can run kubectl get pods -A will show a list of all pods, one of which will begin with 'ingress-azure'.
Get all logs of that pod with kubectl logs <name-of-ingress-controller-pod> . It can be customized as presented above with different parameters.

sai theja 0 Reputation points

2023-01-21T11:37:06.67+00:00

That is the pod, running in the agic namespace.

There are no 400 series or 500 series failed requests on the logs.

Because of this i am unable to retrieve the failed requests.

But on portal, i could see always a failed request.
Sign in to comment