parser

Google’s robots.txt parser is now open source

For 25 years, the Robots Exclusion Protocol (REP) was only a de-facto standard. This had frustrating implications sometimes. On one hand, for webmasters, it meant uncertainty in corner cases, like when their text editor included BOM characters in their robots.txt files. On the other hand, for crawler and tool developers, it also brought uncertainty; for […]

Google open-sources robots.txt parser in push to make Robots Exclusion Protocol an official standard

Google wants to turn the decades-old Robots Exclusion Protocol (REP) into an official internet standard — and it’s making its own robots.txt parser open source as part of the push. The REP, which was proposed as a standard by Dutch software engineer Martijn Koster back in 1994, has pretty much become the standard used by websites

Google open-sources robots.txt parser in push to make Robots Exclusion Protocol an official standard Read More »