Scrapers I block (and allow), with explanationsI've replaced comments in my robots.txt with a more detailed page on scrapers I block.
It includes info on the multiple blocking-approaches and criteria I use, commonly-blocked scrapers I allow, and more fact-checking than most of the more comprehensive alternatives.
twovests wrote
i appreciate this!! i have been sitting on a "set up a robots.txt for my site" and this is a useful guide