Google’s John Mueller On Blocking Robots.txt Files From Being Indexed
Google’s John Mueller recently provided some advice on how to block robots.txt and sitemap files from being indexed in search results. This advice was prompted by a tweet from Google’s Gary Illyes, who randomly pointed out that robots.txt can technically be indexed like any other URL. While it provides special directions for crawling, there’s nothing …
Google’s John Mueller On Blocking Robots.txt Files From Being Indexed Read More »