unsupported

Google is retiring ‘noindex’ and other unsupported robots.txt code

In a new post on their official Webmaster’s Blog, Google announced that it would be officially retiring unsupported lines of code in the robots.txt file that websites use to instruct how the search algorithm should crawl and index their site. It comes after a previous blog post announcing that their production robots.txt parse is going […]

Google is retiring ‘noindex’ and other unsupported robots.txt code Read More »

A note on unsupported rules in robots.txt

Yesterday we announced that we’re open-sourcing Google’s production robots.txt parser. It was an exciting moment that paves the road for potential Search open sourcing projects in the future! Feedback is helpful, and we’re eagerly collecting questions from developers and webmasters alike. One question stood out, which we’ll address in this post: Why isn’t a code