---
layout: post
date: 2025-04-08
link: https://adactio.com/journal/21831
title: And no, a robots txt file doesn’t help
type: linked
cited: Jeremy Keith
---

> If you think these crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots crawl everything they can find, robots.txt be damned.