WordPress 5.3: Robots Meta Tag for Better Search Engine Crawling Prevention

Not every user with a WordPress website is looking to feature on page 1 of Google. In fact, a lot of users would rather that their websites don’t show up on Google or any search engine at all.

And if that’s ever the case, the current WordPress version allows you to do just that by checking the “Discourage search engines from indexing this site” option from the WordPress dashboard Settings > Reading.

There is a small disclaimer underneath it, though:

It is up to search engines to honor this request.

So while this has so far been an easy way to prevent search engines from crawling your website, it’s always possible that your website ends up in the search results.

WordPress 5.3 Will Fix That

The new WordPress version will change the way it prevents search engines from indexing sites. Version 5.3 will drop the robots.txt method and add an updated robots meta tag from keeping your site from getting listed on search engines:

This new meta tag is, of course, a more definitive way to keep your site from getting indexed or crawled.

The current problem in the robots.txt method is well-summarized by Jono Alderson on a trac ticket along with his proposed solution:

The Reading setting infers that it’s intended to prevent search engines from indexing the content, rather than from crawling it. However, the presence of the robots disallow rule prevents search engines from ever discovering the noindex directive, and thus they may index ‘fragments’ (where the page is indexed without content).

Google recently announced that they’re making efforts to prevent fragment indexing. However, until this exists (and I’m not sure it will; it’s still a necessary/correct solution sometimes), we should solve for current behaviors. Let’s remove the robots.txt disallow rule, and allow Google (and others) to crawl the site.

Hopefully, this new meta tag will perform a better job of keeping your development site (or a personal site) from appearing in the search engine.

Do you think this will be a welcome addition to WordPress 5.3? Let us know in the comments section below and we can discuss it in detail!