The message “Indexed Though Blocked by Robots.txt” means your robots.txt file blocked indexing. Usually, this error appears when your website is down, and your robots.txt is modified. For example, if you had server problems, malware or virus attack, etc.
When you test the page in Google Search Console, you will get this:
So this is my story.
My WordPress website broke, and WordPress showed an installation page. When WordPress offers an installation page, you either have a problem with robots.php that is blocking config.php, or the config.php file disappears. In that case, you need to backup the config.php file and robots.txt file, and everything will work again.
Typical Wodpress robots.txt text is:
Copy below:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
However, I uploaded config.php and robots.txt files, but my website XML sitemap was still blocked. I saw this message: “Indexed Though Blocked by Robots.txt.”
To Fix Error Indexed Though Blocked by Robots.txt, you need to upload a new robots.txt file to your server (WordPress site), and then you need to notify Google that your robots.txt file has been changed. To notify Google about the robots.txt change, go to Google Robots Testing Tool.
I thought that Google would do everything automatically, but my new robots.txt file was several days on the server, and Google still used the old one.
As I figure out, google caches your robots.txt file. It would help if you told Google that your robots.txt file had been changed.
After I fixed the problem, my sitemap was again allowed for indexing. In the last step, you need to check once if any URLs are blocked by robots.txt.
Verify the address using the URL.
You can check this out by Coverage > Indexed and then viewing a URL listed there, even though robots.txt is banning it. Then, it will state “No: blocked by robots.txt” for the field Crawl authorized and “Failed: Blocked by robots.txt” for the field Page brought under Crawl. There will be both of these messages in the Crawl section.
Google would not have indexed the URLs under normal circumstances; nonetheless, the search engine uncovered connections to the URLs in question and believed them relevant enough to index. There is a good chance that the fragments shown are not the best possible.
How do you check if a page is blocked by robots.txt?
- Export the URL list and categorize it alphabetically in a separate document by Google Search Console.
- Checked all URLs and Determined whether they included or not
- That needs to be indexed in your database. In this case, you will need to update your robots.txt file to provide Google access to the above URLs.
- It is not your intention for the exploration locomotive to have the right to use. In this situation, the robots.txt file that you use should stay unaltered; however, you should check for any links that should be deleted from the website.
- Search engines may view specific pages, but you do not want them to be indexed. It would help if you had to apply robots no index directives and on your robots.txt file do an update in this particular circumstance.
- No one at any time should have access to this information. Take, for instance, the concept of a staging environment.
- If it is unclear which portion of your robots.txt is initiating these URLs to be prohibited, pick a URL, and in the right-hand window, click the TEST ROBOTS.TXT BLOCKING button. This will allow you to see which part of your robots.txt is causing the problem. This will cause a new window to appear, showing the line in robots.txt that checks Google from viewing the URL in question.
- When you have made changes, you should click the VALIDATE FIX button to ask Google to review your robots.txt file about your URLs.
Indexed but banned by WordPress robots.txt modification
The approach for resolving this problem for WordPress sites is the same as that detailed in the preceding phases. However, for locating your robots.txt file in WordPress, below are a few tips:
WordPress combined with All in One SEO
If you’re using the All in One SEO plugin, change your robots.txt file as shown below:
- Enter your wp-admin credentials.
- Navigate to All in One SEO > Robots.txt in the sidebar.
WordPress plus Rank Mathematics
If you’re using the Rank Math SEO plugin, change your robots.txt file as shown below:
- Log in to your wp-admin area.
- Navigate to Rank Math > General Settings in the sidebar. Click Edit robots.txt.
WordPress combined with Yoast SEO
If you’re using the Yoast SEO plugin, edit your robots.txt file as shown below:
- Log in to your wp-admin area.
- Navigate to Yoast SEO plugin > Tools on the sidebar.
- Go to file editor.
Pro tip
If you’re functioning on a WordPress site that has not yet gone live and can’t figure out why your robots.txt file includes the following:
User-agent: *
Disallow:
At that moment, go to Settings > Reading and search for Search Engine Visibility.
If the option Discourage search engines from indexing this site is selected, WordPress will build a robots.txt file that prohibits search bars from visiting the site.
Shopify pages are indexed but are prevented by a robots.txt fix.
Shopify system does not allow you to accomplish your robots.txt; thus, you’re dealing with a universally-applicable one.
You may have comprehended the “Indexed, but banned by robots.txt” gotten an email from Google titled “Innovative index reporting problem found” or a statement in Google Search Console about this issue. In SEO, you should never quit anything unplanned; thus, we advise that you constantly verify which URLs this pertains to.
Examine the URLs to see whether any essential URLs are banned. If so, you have two solutions that involve some effort but enable you to modify your Shopify robots.txt file:
- Configure a reverse proxy
- Use Cloud flare Workers
Whether or whether you should pursue these possibilities rests on the prospective payoff. If it is substantial, consider executing any of these choices.
On Square space’s platform, you may use the same strategy.
Information That Is Frequently Requested
1) Why does Google show this error warning for each of my pages?
Because robots.txt contains instructions to block access to some sites, Google detected links to those sites. Google will index these pages once it thinks they are significant enough to warrant attention.
2) What steps will you take to address this oversight?
You need to ensure that the websites you need Google to index are accessible to its spiders for them to be indexed. Also, you should avoid creating internal connections to other websites, especially if you do not want such websites to be indexed. The detailed solution may be found in the part of this post titled “How to repair ‘Indexed, but banned by robots.txt.'”
3) Is altering the robots.txt file stored in my WordPress installation feasible?
You may change robots.txt straight from the wp-admin dashboard if you use a popular SEO plugin like Yoast, Rank Math, or All in one SEO. These plugins are available.
Learn more about authors in Nimblefreelancer's team biography page.
