For some reason, since Friday, none of the websites on my VM are allowing Google Bot to see their robots.txt files, I have made no changes to the server, there have been no updates to Jugganaut WAF, etc.

Anyone have any idea what might be causing the issue?

This is my current robots.txt file contents:

User-agent: *

Allow: /

Disallow: /tour/*

Disallow: /precision-micro-360-virtual-tour/*

User-agent: Googlebot

User-agent: AdsBot-Google

Allow: /

Sitemap: https://www.precisionmicro.com/sitemap_index.xml

WP Defender - Begin

User-agent: *

Disallow: c9fd19c439ba1bb1

WP Defender - End

There is nothing in Enterprise applications.The VM is not behind Azure WAF.

It is a Linux box, not running IIS.

There are no bot blocking policies in Juggernaut

Any other ideas?

This started on Friday, I made no changes.

Hello Jonny Wilde

Open private browser window and navigate to test rich results shared details via private messages.

Test with a Command Line Tool: Use curl or wget from a machine outside your network. Added details via private messages.

Look for a 200 OK status code and if you get a 4xx (403 Forbidden, 404 Not Found) or 5xx error, the issue is with your web server's configuration or file permissions.

Google's Rich Results Test: Enter your robots.txt URL. It will show you whether Google can fetch it and content it receives.

Go to the URL Inspection tool for your domain in Search Console. Enter the URL of your robots.txt file. Click TEST LIVE URL and this will show exact HTTP response code and content Google's crawler is receiving right now.

Your robots.txt content shows you are using WP Defender, a WordPress security plugin.

Security plugins often update their firewall or blocklist rules automatically.

Log into your WordPress admin dashboard. Go to the WP Defender settings. Look for sections named Firewall, IP Blocking, Country Blocking, or Advanced Blocking.

There should be a log of blocked requests. Look for any entries around the time of the issue that show Googlebot's IP addresses being blocked. Google publishes its crawler IP ranges here. As a test, temporarily disable the WP Defender firewall.

Immediately after disabling, re-run the tests from Google rich results test If Googlebot can now access the file, you have found the cause. You can then re-enable it and configure an allow rule for Googlebot's user-agent or IP ranges. Even though you have said there are no bot policies, please double-check.

A rule might have been updated or falsely flagging Googlebot. Review the logs for requests to robots.txt.

htaccess File (Apache) or Server Config (Nginx)> A misconfiguration here could be blocking access.

Apache > Check here for any rules that might block access to robots.txt based on user-agent. A rule intended to block bad bots might be too aggressive.

Nginx: Check server block configuration files for any location directives that might interfere with serving the robots.txt file

In the Azure Portal, go to your VM's networking settings and review the associated Network Security Group (NSG). Ensure there are no inbound rules that could be blocking traffic from Google's IP ranges. And if you have Azure DDoS Network Protection enabled on the VNet, review alerts.

On your Linux server, please ensure the robots.txt file and the directories above it have read permissions for the user that the web server process example nginx runs under.

bash

Navigate to the web root directory and run :

ls -la robots.txt

I have deactivated and deleted Defender, there is also Wordfence WAF but it is in learning mode, not blocking. And there are no rules created.

There are no mentions of Google bot from after Friday last week in the Juggernaut access logs on the server. .htaccess is bellow:

BEGIN WP Rocket

Use UTF-8 encoding for anything served text/plain or text/html

AddDefaultCharset UTF-8

Force UTF-8 for a number of file formats

<IfModule mod_mime.c>

AddCharset UTF-8 .atom .css .js .json .rss .vtt .xml

</IfModule>

FileETag None is not enough for every server.

<IfModule mod_headers.c>

Header unset ETag

</IfModule>

Since we’re sending far-future expires, we don’t need ETags for static content.

developer.yahoo.com/performance/rules.html#etags

FileETag None

<IfModule mod_alias.c>

<FilesMatch ".(html|htm|rtf|rtx|txt|xsd|xsl|xml)$">

<IfModule mod_headers.c>

Header set X-Powered-By "WP Rocket/3.19.4"

Header unset Pragma

Header append Cache-Control "public"

Header unset Last-Modified

</IfModule>

</FilesMatch>

<FilesMatch ".(css|htc|js|asf|asx|wax|wmv|wmx|avi|bmp|class|divx|doc|docx|eot|exe|gif|gz|gzip|ico|jpg|jpeg|jpe|json|mdb|mid|midi|mov|qt|mp3|m4a|mp4|m4v|mpeg|mpg|mpe|mpp|otf|odb|odc|odf|odg|odp|ods|odt|ogg|pdf|png|pot|pps|ppt|pptx|ra|ram|svg|svgz|swf|tar|tif|tiff|ttf|ttc|wav|wma|wri|xla|xls|xlsx|xlt|xlw|zip)$">

<IfModule mod_headers.c>

Header unset Pragma

Header append Cache-Control "public"

</IfModule>

</FilesMatch>

</IfModule>

<IfModule mod_mime.c>

AddType image/avif                                  avif

AddType image/avif-sequence                         avifs

</IfModule>

Expires headers (for better cache control)

<IfModule mod_expires.c>

ExpiresActive on

ExpiresDefault                              "access plus 1 month"

# cache.appcache needs re-requests in FF 3.6 (thanks Remy ~Introducing HTML5)

ExpiresByType text/cache-manifest           "access plus 0 seconds"

# Your document html

ExpiresByType text/html                     "access plus 0 seconds"

# Data

ExpiresByType text/xml                      "access plus 0 seconds"

ExpiresByType application/xml               "access plus 0 seconds"

ExpiresByType application/json              "access plus 0 seconds"

# Feed

ExpiresByType application/rss+xml           "access plus 1 hour"

ExpiresByType application/atom+xml          "access plus 1 hour"

# Favicon (cannot be renamed)

ExpiresByType image/x-icon                  "access plus 1 week"

# Media: images, video, audio

ExpiresByType image/gif                     "access plus 4 months"

ExpiresByType image/png                     "access plus 4 months"

ExpiresByType image/jpeg                    "access plus 4 months"

ExpiresByType image/webp                    "access plus 4 months"

ExpiresByType video/ogg                     "access plus 4 months"

ExpiresByType audio/ogg                     "access plus 4 months"

ExpiresByType video/mp4                     "access plus 4 months"

ExpiresByType video/webm                    "access plus 4 months"

ExpiresByType image/avif                    "access plus 4 months"

ExpiresByType image/avif-sequence           "access plus 4 months"

# HTC files  (css3pie)

ExpiresByType text/x-component              "access plus 1 month"

# Webfonts

ExpiresByType font/ttf                      "access plus 4 months"

ExpiresByType font/otf                      "access plus 4 months"

ExpiresByType font/woff                     "access plus 4 months"

ExpiresByType font/woff2                    "access plus 4 months"

ExpiresByType image/svg+xml                 "access plus 4 months"

ExpiresByType application/vnd.ms-fontobject "access plus 1 month"

# CSS and JavaScript

ExpiresByType text/css                      "access plus 1 year"

ExpiresByType application/javascript        "access plus 1 year"

</IfModule>

Gzip compression

<IfModule mod_deflate.c>

Active compression

SetOutputFilter DEFLATE

Force deflate for mangled headers

<IfModule mod_setenvif.c>

<IfModule mod_headers.c>

SetEnvIfNoCase ^(Accept-EncodXng|X-cept-Encoding|X{15}|~{15}|-{15})$ ^((gzip|deflate)\s*,?\s*)+|[X~-]{4,13}$ HAVE_Accept-Encoding

RequestHeader append Accept-Encoding "gzip,deflate" env=HAVE_Accept-Encoding

Don’t compress images and other uncompressible content

SetEnvIfNoCase Request_URI \

.(?:gif|jpe?g|png|rar|zip|exe|flv|mov|wma|mp3|avi|swf|mp?g|mp4|webm|webp|pdf)$ no-gzip dont-vary

</IfModule>

</IfModule>

Compress all output labeled with one of the following MIME-types

<IfModule mod_filter.c>

AddOutputFilterByType DEFLATE application/atom+xml \

	                          application/javascript \

	                          application/json \

	                          application/rss+xml \

	                          application/vnd.ms-fontobject \

	                          application/x-font-ttf \

	                          application/xhtml+xml \

	                          application/xml \

	                          font/opentype \

	                          image/svg+xml \

	                          image/x-icon \

	                          text/css \

	                          text/html \

	                          text/plain \

	                          text/x-component \

	                          text/xml

</IfModule>

<IfModule mod_headers.c>

Header append Vary: Accept-Encoding

</IfModule>

</IfModule>

END WP Rocket

Redirect 301 /de/aetzen/aetzprozess /de/aetzen/aetztechnik

Redirect 301 /fr/qu-est-ce-que-la-gravure-chimique-et-pourquoi-devrais-je-l-utiliser/ /fr/decoupe-chimique/processus-de-decoupe-chimique/

Redirect 301 /de/was-ist-chemisches-aetzen-und-warum-sollte-ich-es-verwenden/ /de/aetzen/aetztechnik/

BEGIN WordPress

The directives (lines) between "BEGIN WordPress" and "END WordPress" are

dynamically generated, and should only be modified via WordPress filters.

Any changes to the directives between these markers will be overwritten.

<IfModule mod_rewrite.c>

RewriteEngine On

RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]

RewriteBase /

RewriteRule ^index.php$ - [L]

RewriteCond %{REQUEST_FILENAME} !-f

RewriteCond %{REQUEST_FILENAME} !-d

RewriteRule . /index.php [L]

</IfModule>

END WordPress

Block direct access to wp-cron.php

<Files "wp-cron.php">

Order Deny,Allow

Deny from all

# Only allow server-side requests

Allow from 127.0.0.1

Allow from ::1

</Files>

Wordfence WAF

<Files ".user.ini">

<IfModule mod_authz_core.c>

Require all denied

</IfModule>

<IfModule !mod_authz_core.c>

Order deny,allow

Deny from all

</IfModule>

</Files>

END Wordfence WAF

BEGIN Headers Security Advanced & HSTS WP 5.0.44

<IfModule mod_headers.c>

Header set Access-Control-Allow-Methods "GET,POST"

Header set Access-Control-Allow-Headers "Content-Type, Authorization"

Header set Content-Security-Policy "upgrade-insecure-requests;"

Header set Cross-Origin-Embedder-Policy "unsafe-none; report-to='default'"

Header set Cross-Origin-Embedder-Policy-Report-Only "unsafe-none; report-to='default'"

Header set Cross-Origin-Opener-Policy "unsafe-none"

Header set Cross-Origin-Opener-Policy-Report-Only "unsafe-none; report-to='default'"

Header set Cross-Origin-Resource-Policy "cross-origin"

Header set Permissions-Policy "accelerometer=(), autoplay=(), camera=(), cross-origin-isolated=(), display-capture=(self), encrypted-media=(), fullscreen=, geolocation=(self), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), payment=, picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), xr-spatial-tracking=(), gamepad=(), serial=()"

Header set Referrer-Policy "strict-origin-when-cross-origin"

Header set Strict-Transport-Security "max-age=63072000"

Header set X-Content-Security-Policy "default-src 'self'; img-src *; media-src * data:;"

Header set X-Content-Type-Options "nosniff"

Header set X-Frame-Options "SAMEORIGIN"

Header set X-Permitted-Cross-Domain-Policies "none"

</IfModule>

END Headers Security Advanced & HSTS WP

END OF HTACCESSS

nginx.conf file:

These are the only rules with Deny, all full port denies, nothing with IP ranges.

I do not have Azure DDoS Network Protection.

It is done with Juggernaut on the server.

ls -la robots.txt

-rwxr-xr-x. 1 charlottenburgas psacln 270 Sep 17 13:18 robots.txt

All sorted now, it was my firewall, not the VM.

Hi Jonny Wilde,

Thanks for the confirmation. Let me know if you require any additional assistance from my end. I am happy to help you with the queries.

Thanks,

Rashmika

Hi Jonny Wilde

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

We need to Verify Current Robots.txt Content

https://yourdomain.com/robots.txt (access your robots.txt file directly)

If you see this then azure Application Proxy is the cause:

please refer the Microsoft doc: https://learn.microsoft.com/en-us/entra/identity/app-proxy/application-proxy-security

User-agent: *

Disallow: /

Check for Azure Application Proxy:

1. Log into the Azure Portal
2. Navigate to Azure Active Directory > Enterprise applications
3. Check if applications configured with Application Proxy
4. Also Check if your websites are published through Application Proxy https://techcommunity.microsoft.com/discussions/azure/azure-app-proxy---blocked-bots/2232128

Allow access to robots.txt in Azure Web application Firewall : https://stackoverflow.com/questions/78098607/allow-access-to-robots-txt-in-azure-web-application-firewall

Verify IIS Configuration

1. Open IIS Manager
2. Check for SEO Toolkit installation under server features.
3. Look for any HTTP managing robots.txt requests
4. Review application configuration for automated robots.txt

reference Microsoft documentation for managing robots.txt and sitemap Files

https://learn.microsoft.com/en-us/iis/extensions/iis-search-engine-optimization-toolkit/managing-robotstxt-and-sitemap-files

Check once WAF Settings

Access Jugganaut WAF management interface

Check for recent rule changes or updates

1. Look for bot blocking policies that are active and review logs for any automatic policy updates

Azure Web Application Firewall- Bot Manager Scenarios: https://techcommunity.microsoft.com/blog/azurenetworksecurityblog/azure-web-application-firewall--bot-manager-scenarios/3855731

Kindly let us know if the suggested steps helps or you need further assistance on this issue.

Regards

Himanshu