Knowledge base v2.13.0 · Community edition · data through 2 October 2026
Day 1 · Wednesday 30 September 2026 · 15:30
One community talk was recorded: robots.txt pitfalls seen in the wild (Dave Smart, Tech SEO, Tame the Bots: the host's hand-over, his self-introduction and the official speaker list). The audio recording ends during his talk.
Dave Smart said robots.txt groups are not additive: if a group matches a crawler's user agent, only that group's rules apply, so a googlebot group that blocks /dogs/ leaves Googlebot free to crawl the /goats/ and /cows/ that the * group blocks.
Speaker Dave SmartEvidence transcript
Used byrequirement DEV-SRV-10glossary term User-agent group
Dave Smart said robots.txt is checked for every URL in a redirect chain and crawling stops at the first blocked one; in his example a site redirected through /cart/ with JavaScript to set the local currency and back, and because /cart/ was disallowed the page was reported as blocked.
Speaker Dave SmartEvidence transcript
Used byrequirement DEV-CAN-11glossary term Redirect chain
Dave Smart said Search Console reports such a block only on the first URL of the chain, which is confusing: the URL shown as blocked by robots.txt is not itself disallowed.
Speaker Dave SmartEvidence transcript
Used byrequirement DEV-CAN-11glossary term Redirect chain
Dave Smart said this applies to all redirects, not only JavaScript ones; his examples: a redirect through an external authorisation service that is blocked by its own robots.txt, content that moved through several URLs over the years with one of them later blocked, and unexpected redirects, such as one served only to Googlebot's user agent.
Speaker Dave SmartEvidence transcript
Used byrequirement DEV-CAN-11
When Search Console says a URL is blocked by robots.txt but the file does not block it, check whether the URL redirects and test every URL in the chain.
Speaker Dave SmartEvidence transcript
Used byrequirement DEV-CAN-11
Dave Smart said robots.txt groups are not additive: if a group matches a crawler's user agent, only that group's rules apply, so a googlebot group that blocks /dogs/ leaves Googlebot free to crawl the /goats/ and /cows/ that the * group blocks.
Rules can be grouped for several crawlers by repeating user-agent lines. A crawler follows only the most specific group that names it.