---

layout: post
date: 2025-04-08
link: https://adactio.com/journal/21831
title: And no, a robots txt file doesn’t help
cited: Jeremy Keith

---

> If you think these crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots crawl everything they can find, robots.txt be damned.