Application Gateway returning 403

Gregory Smith 20

I have a container app env setup with 2 container apps setup.

I have an Application Gateway in front of them with a listener rule with path based routing to both in their own backend pool.

The idea is to have a single rule as the "catch all" and then path based routing to route to the other backends, but even though the insights of the application gateway shows as healthy, anytime I hit the public endpoint, I get a 403. I can verify that each of the container apps is responding when hit directly, but the Application gateway just sends back 403s.

(I'm trying to port an existing ECS Fargate application from AWS over to Azure, so some of this is slightly different if it's a simple config that I missed)

KapilAnanth-MSFT 44,921 Reputation points Microsoft Employee

2023-08-31T04:48:49.0566667+00:00

@Gregory Smith

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you are receiving 403 from Application gateway.

First you have to make sure that the error is coming from App gateway and not from the backend itself.

Please enable Azure App gateway logs

Refer : https://learn.microsoft.com/en-us/azure/web-application-firewall/ag/web-application-firewall-logs

Once enable, reproduce the error and check the Access Logs

For v2,

"serverStatus" and "httpStatus" will tell us the response from backend and App GW respectively.

Can you please share a screenshot of the log of one such 403 response?

Cheers,

Kapil
Gregory Smith 20 Reputation points

2023-08-31T10:38:56.64+00:00

I did turn off the firewall just to be sure and also checked the access log and I do see the 403 in the access log, but don't see any other information as to why it issues the 403.

If I hit the endpoint of the container app directly, it works. But I'm wanting to use the path base routing of the application gateway.
KapilAnanth-MSFT 44,921 Reputation points Microsoft Employee

2023-08-31T11:15:09.17+00:00
@Gregory Smith

Can you confirm if you are using App gateway v2?

Can you please share what was the field value for "serverStatus"

If you are using v1,

Please share the entire message in "RequestQuery"

Cheers,

Kapil
Gregory Smith 20 Reputation points

2023-08-31T13:00:53.8233333+00:00

serverStatus was also 403.

The backend pool seems healthy from the probes.

My goal is to have a single listener pass everything to the client UNLESS a path match takes it to another backend endpoint, I'm not sure quite how to figure that, so for now I just have both enpoints listed, one with a /api/* path and the client with a /* path.

This is the insights of the App Gateway:
Gregory Smith 20 Reputation points

2023-08-31T13:02:09.2933333+00:00

Tier is WAFv2 if that answers the v1/v2 question from above.
KapilAnanth-MSFT 44,921 Reputation points Microsoft Employee

2023-08-31T13:08:24.17+00:00
@Gregory Smith

If the server is giving 403, App gateway has very little play here.

App gateway just sends the response from Backend service to the client.

The health probes being healthy is not an exact indicator.

Let's say the health probes are probing "www.yoursite.com/healthcheck.txt" or just "www.yoursite.com"

This returns 200 and thus the backend is considered healthy.

But when you access "www.yoursite.com/pagethatgives403.txt" - it might give a 403 response and App GW just sends it to the client.

Instead of troubleshooting from the App gateway perspective, I would suggest you to check the backend (containerApps) as to why it is giving a 403 response for requests originating from App Gw.

This should be easy to filter as you can either

Check for 403 responses

or Filter using App GW subnet's IP range

Cheers,

Kapil
KapilAnanth-MSFT 44,921 Reputation points Microsoft Employee

2023-09-05T06:11:24.45+00:00

@Gregory Smith

Reaching out to check if there are any questions on this.

Please let us know if we can be of any further assistance here.

Thanks,

Kapil
Gregory Smith 20 Reputation points

2023-09-05T09:59:38.66+00:00

I was never able to solve this completely, but I think I did narrow it to being a certificate issue on the backend. This was for a POC to get the existing application running in Azure. Between this and a few other issues, it's going to take a little Dev time to get the application to play nice, so the env has been destroyed already until there's time for devs to work on this. Thanks for the quick responses.
KapilAnanth-MSFT 44,921 Reputation points Microsoft Employee

2023-09-05T10:13:42.5666667+00:00

@Gregory Smith

Thanks for your continued contribution on Q&A and appreciate much for taking the time to share your feedback.

Sure, you can reach out to us once you are available to proceed further.

Cheers,

Kapil

Accepted answer

msrini-MSFT 9,281 Reputation points Microsoft Employee

2023-09-02T04:30:51.7766667+00:00
Hi,

403 is a client side error. If you are able to see that 403 is sent by Application Gateway, that means your backend is responding with 403. To understand why it responds with this 403, I would like the following info from your end:

Fiddler tracer when you access your site via Application Gateway

Screenshots of your Application Gateway settings - Probe, HTTP Settings, Rule, Listener, Backend Health Status

This can help us understand how Application Gateway would be sending the request to your backend and we can figure out why your backend is responding with 403.

Regards,

Karthik Srinivas
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Application Gateway returning 403

0 additional answers

Your answer