Website not getting crawled

Dear Martin,

I am having the same issue. This has been ongoing for a couple months. Tickets opened to Bing Webmasters are not being responded to. No one is helping. All checks have been performed and there are no issues on the website that would prevent indexing. The site is even ranked #1 on Google for the related keyword search. There is clearly some kind of bug going on, and I have tried informing the Webmaster team through numerous tickets. The scans come back without any issues and the reports even claim the site is indexed, but it is not indexed. Why is Microsoft ignoring tickets and not addressing this problem?

Dear Jdub26,

Welcome to the Microsoft Community

Thank you for reaching out with your concerns about your website not being indexed by Bing and Google. It can be frustrating when you're following best practices but not seeing the desired results.

You mentioned receiving an error indicating that "Bing is aware of the URL but was unable to index it due to some issue." This message typically implies that while Bingbot has found your site, there are certain reasons preventing it from successfully indexing the pages. Let's go through a series of steps to diagnose and potentially resolve these issues.

1. Review Bing Webmaster Guidelines. Ensure that your site complies with the Bing Webmaster Guidelines. These guidelines offer essential practices that improve your site's chances of being indexed.

2. Check for Crawl Errors. Bing Webmaster Tools is an invaluable resource for identifying crawl errors.

• Go to Bing Webmaster Tools.
• Log in to your account and select your site.
• Navigate to the Reports & Data > Crawl Information section.
• Here, you'll find detailed reports about any crawl errors, such as 404 errors, server errors, or blocked content.

3. Ensure Accessibility. Make sure that Bingbot can access and crawl your site’s content.

• Robots.txt File: Verify that your robots.txt file is not inadvertently blocking Bingbot. Ensure the following line is present to allow crawling:

User-agent: bingbot

     Disallow:

• Sitemap Submission: Submit an up-to-date sitemap to Bing to help the crawler understand the structure of your site. You can do this through the Sitemaps section in Bing Webmaster Tools.

4. Check for Technical Issues. Various technical factors can prevent Bing from indexing your site correctly.

• Ensure Correct HTTP Status Codes: Check that your pages return the correct HTTP status codes. For example, ensure that pages return 200 OK and not 404 Not Found or 500 Internal Server Error.
• Optimize Site Performance: Slow-loading pages or server timeouts can adversely affect crawling. Use tools like Google PageSpeed Insights or GTmetrix to assess and improve site performance.
• Disclaimer: Microsoft provides no assurances and/or warranties, implied or otherwise, and is not responsible for the information you receive from the third-party linked sites or any support related to technology.

5. Content Quality and Relevance. Bing prioritizes high-quality, relevant content.

• Ensure Original, High-Quality Content: Avoid duplicate content. Ensure that your site provides valuable and original information that adheres to Bing’s content standards.
• Optimize Meta Tags: Ensure that each page has unique and descriptive title tags and meta descriptions. These help Bing understand the content and purpose of each page.

If after following these steps you continue to experience issues, consider reaching out directly to Bing Webmaster Support. They may provide specific insights or solutions based on your site’s situation.

Addressing the factors above should significantly improve your chances of having your site indexed by Bing. I hope these steps guide you towards resolving the issue. Please do not hesitate to reach out if you have any further questions or need additional assistance.

Best Regards,

Martin | Microsoft Community Support Specialist