directives

Bing Says They Ignore Default Robots Directives If There Is A Bingbot Section

Frédéric Dubut from Bing’s search team said on Twitter that if you create a specific robots.txt directive for Bingbot, their crawler, then Bing will only look at that specific section. So you should make sure that when you do that, copy all the directives from the default to the Bingbot section that you want Bing […]

Robots.txt tip from Bing: Include all relevant directives if you have a Bingbot section

SEO BLOG / Good To SEO

Frédéric Dubut, a senior program manager at Microsoft working on Bing Search, said on Twitter Wednesday that when you create a specific section in your robots.txt file for its Bingbot crawler, you should make sure to list all the default directives in that section. Specify directives for Bingbot. “If you create a section for Bingbot

Robots.txt tip from Bing: Include all relevant directives if you have a Bingbot section Read More »