Debugging 403 Ip Forbidden errors

Daniel Becroft 26

We have some Azure WebApps (AppServices) deployed, and have implemented the IP address restrictions so that only traffic from a defined set of networks is allowed through.

Periodically, however, we are receiving HTTP 403 Ip Forbidden error messages on the AppService, even when access was previously allowed through, and neither the firewall rules, or our outgoing IP address, has changed.

It seems to come and go (we had one that was failing yesterday, but was working this morning). Deleting and recreating the webapp seems to resolve the issue, as well as removing all the IP address restrictions.

Is there a way to debug these errors? I can't see the responses in the Log Stream or any other item in the AppService blade, so it would be good to find somewhere that says "Request from IP x.x.x.x was blocked due to restrictions" or similar.

> curl -I https://xxxxxx-xxxxxx-xxxxxx-315-xxxxxx-api.azurewebsites.net/api/Clients?status=Active
HTTP/1.1 403 Ip Forbidden
Content-Length: 2345
Content-Type: text/html
Server: Microsoft-IIS/10.0
Date: Fri, 07 Feb 2020 04:27:00 GMT

The apps are deployed as Linux containers, running on a B1 service plan.

I have seen reference that this error could come about due to quotas being exceeded, but I can't see anything that refers to an exceeded quota (everthing appears to be within the allowed limits).

Suwat Bodin 36 Reputation points Microsoft Employee

2020-02-10T04:27:32.72+00:00

See if the debugging section in this link helps. https://github.com/madsd/articles/wiki/Access-restriction-X-Forward-headers-(preview)
Daniel Becroft 26 Reputation points

2020-02-10T07:25:09.56+00:00

Thanks @Suwat Bodin , I've verified that the external IP being detected by Azure is the correct one, and that it is correctly added in the firewall rules.
Jeremy Brooks 572 Reputation points Microsoft Employee

2020-02-21T20:34:04.34+00:00

Try to see if this helps in checking to see if the IP is blocked due to IP restrictions. If the request is being blocked by IP restrictions it occurs on the FrontEnds of the infrastructure so the 403s would not be seen in the application logs. There's a detector available in the Azure Portal to confirm what IPs are being blocked if that is the cause, see below.

The access restrictions capability is implemented in the App Service front-end roles, which are upstream of the worker hosts where your code runs. Therefore, access restrictions are effectively network ACLs.

https://learn.microsoft.com/en-us/azure/app-service/app-service-ip-restrictions
Stuart Kemp 1 Reputation point

2022-08-30T13:12:03.677+00:00

Is there any way to see more detailed logs on this, like the time of each request for example?

I have a service that reports some rejections due to our IP restrictions but, when I look in the ApplicationInsights connected to the app, the only records in the requests table are 200s or 404s.
I'm assuming the AppInight logs happen as the Application level, while the IP rejections are happening at the App Service level, so don't get logged in the same place. But Azure must be logging them somewhere if it can retrieve the blocked IPs and their request count, right?
Ryan Hill 26,136 Reputation points Microsoft Employee

2022-09-01T03:41:03.573+00:00

Try using AppServiceIPSecAuditLogs log data under Diagnostic Settings.

1 answer

Venkatakrishnan Damodaran 1 Reputation point

2020-02-19T18:42:01.15+00:00

Are you try to enable Web Server Logging and check?

Click App Service Logs -> Click File System in Web server logging. Also enabled Failed request tracing.

After the above settings. you will be able to see the traffic in Kudu Console via Advanced Tools-> Log directory.
Please sign in to rate this answer.
Daniel Becroft 26 Reputation points

2020-02-21T00:55:45.13+00:00

Thanks. I've checked the logs, and there's nothing appearing. The problem is that Azure is blocking the request before it gets anywhere near my container. There's nothing in any of the log files (that I can find) where Azure is recording a HTTP 403 IP Forbidden response.

Venkatakrishnan Damodaran 1 Reputation point

2020-03-17T01:35:29.613+00:00

Hi daniel,

What kind of firewall you are using. Are you enabling IP restrictions via web.config or you have front-end WAF?. If you have WAF. I have faced several issues with WAF for false positive traffic. you have to enable Logging for firewall via Storage account. Click "Diagnostic settings" in the WAF service, Click Diagnostic Settings-> provide name and select necessary log details and provide storage account. After that the all the traffic going through firewall logged in the storage account as a json file for each hour. you will be able to investigate that json file for any firewall false positive access denied issues. let me know how it goes.

Thanks for your patience.

ajkuma 22,521 Reputation points Microsoft Employee

2020-04-21T19:35:08.427+00:00

Daniel, Apologies for the delay. Just checking in to see if you have had a chance to see the previous post. Kindly share the requested information to better assist you.

Daniel Becroft 1 Reputation point

2020-04-21T22:37:02.39+00:00

Sorry @ajkuma . I don't have a separate WAF resource in front of the web app.

The solution was provided in a comment by another poster: using the "Diagnose and Solve problems", there's an option to check configuration and it will show the masked IP addresses that are being blocked. This is sufficient for our requirement.s

ajkuma 22,521 Reputation points Microsoft Employee

2020-06-05T04:08:18.163+00:00

Thanks for sharing the solution that worked for you.
Kindly "Accept" the answer on the post which helped you to benefit the community.
Sign in to comment