Azure Application Gateway
An Azure service that provides a platform-managed, scalable, and highly available application delivery controller as a service.
1,217 questions
how to fixAzure applications have been getting occasional 403 (“Forbidden”) errors.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(124.80000000000001, 31%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVA%3C/text%3E%3C/svg%3E)
VP, Anand
0
Reputation points
Over the last month Azure applications have been getting occasional 403 (“Forbidden”) errors.
• The errors always seem to happen during the initial Azure AD OAUTH2 authentication phase (where users are redirected to login.microsoftonline.com and back) – though I’ve not been able to trap a failure “in the wild” to see exactly the step that is failing. Once the user is authenticated, all works as expected.
• The errors occur only occasionally, but do seem to be clumped together in time
• They occur in at least two of our subscriptions (Test and Prod) and across multiple application technologies (PHP and .NET). Test, in particular, has a low usage so it doesn’t feel like a resource exhaustion problem within the subscription
It’s conceivable it could be a problem at our end (maybe with our application gateway), but we’ve made no changes. At the moment, it feels more like a more central problem. Any ideas from your end?
- All hrefs/redirects point to the app gateway
- The problem is intermittent. Most of the time things work fine; only occasionally do things fail (using exactly the same calls as work normally)
- The forbidden IP address is that of the app gateway itself. So traffic must be coming from there!
Azure Application Gateway
{count} votes
-
KapilAnanth-MSFT 49,611 Reputation points Microsoft Employee Moderator2023-10-10T04:56:41.0566667+00:00 Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.
I understand that you are facing 403 Errors with your Application gateway.
- It is highly possible that this 403 Errors are coming from your backend and the App gateway is simply forwarding it.
- Did you check your backend application logs for any 403?
- May I ask if you have enabled WAF?
The best way to figure out the culprit is to enable logging in Azure Application gateway.
- Enable Logging : https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-diagnostics#enable-logging-through-the-azure-portal
- After enabling, it may take upto an hour for the logs to reflect
- Please check the Access Logs and Firewall log
After querying the Access logs,
- If you see "serverStatus" as 403, it means the error is from the backend
- In this case, you must troubleshoot the backend and not the application gateway
- If the "httpStatus" is 403 provided "serverStatus" is something else or blank, then you should start querying Firewall logs.
AzureDiagnostics | where ResourceProvider == "MICROSOFT.NETWORK" and Category == "ApplicationGatewayAccessLog"In the Firewall Logs,
- Find the request that got a 403.
- See what is the "ruleSetType", "ruleId" and "ruleSetVersion"
AzureDiagnostics | where ResourceProvider == "MICROSOFT.NETWORK" and Category == "ApplicationGatewayFirewallLog"P.S : You can use the field "transactionId" which is present in both Firewall logs and Access logs to correlate logs from both
Cheers,
Kapil
-
VP, Anand 0 Reputation points
2023-10-10T07:34:17.8233333+00:00 - May I ask if you have enabled WAF?
yes we have enabled WAF in the prod and disable WAF in the test .Detection only. Also, we tried turning it off but it made no difference.
The problem is intermittent. Most of the time things work fine; only occasionally do things fail (using exactly the same calls as work normally).
The forbidden IP address is that of the app gateway itself. So traffic must be coming from there.
The 403 errors do indeed come from the backend app. They are complaining about a forbidden IP address. The specific forbidden IP address they are complaining about is the application gateway's public IP. (So, in the case of non-prod, it getting non prod public ip )
The app gateway, needless to say, is in the same subscription as the backend app. It has both a public IP and a private IP. And for some reason, the public IP is occasionally (just occasionally) passed through to the backend app, and hence rejected.
-
KapilAnanth-MSFT 49,611 Reputation points Microsoft Employee Moderator
2023-10-10T08:32:07.2933333+00:00 I take it that you are using both Public and private IP
- May I ask what is the backend ?
- Is it a VM or an App Service?
- Generally, the Public IP of the App gateway will be passed only if the backend is a Publicly reachable service.
- Is it a VM or an App Service?
- App gateway will only send a request using it's Public IP if the backend is a public endpoint.
Cheers,
Kapil
- May I ask what is the backend ?
-
VP, Anand 0 Reputation points
2023-10-10T09:02:05.0566667+00:00 its An app service.
The app service is not publicly reachable. its access Via private endpoints
-
KapilAnanth-MSFT 49,611 Reputation points Microsoft Employee Moderator
2023-10-10T09:15:35.1733333+00:00 In this case, the App gateway instances would reach to the App Service via the private IP of the subnet in which the App gateway is deployed.
- Are you using a custom DNS server?
- In this case, it is possible that the DNS server resolved to the Public IP of the App service.
- Can you please share the logs from App Service where the Public IP of the App gateway tried to access it?
- Without this logs, we will not be able to isolate the root cause.
- Also, please share the logs from the App gateway that resulted in the 403 from backend.
- Since you have mentioned, the error is highly intermittent, you can use the timestamp to cross reference the logs
Cheers,
Kapil
- Are you using a custom DNS server?
-
KapilAnanth-MSFT 49,611 Reputation points Microsoft Employee Moderator
2023-10-11T05:51:24.25+00:00 May I know if you got a chance to review my previous comment?
Please let me know if you are facing any challenges or if there are any follow-up questions, I shall be glad to address them.
Thanks,
Kapil
-
KapilAnanth-MSFT 49,611 Reputation points Microsoft Employee Moderator
2023-10-12T03:52:54.9166667+00:00 Can you please update us if the action plan provided was helpful?
Should there be any follow-up questions or concerns, please let us know and we shall try to address them.
Thanks,
Kapil
Sign in to comment
Sign in to answer