And no, a robots txt file doesn’t help

If you think these crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots crawl everything they can find, robots.txt be damned.

» Jeremy Keith →