Submitted by nitori in technology
twovests wrote
Reply to comment by nitori in you don't need anubis by nitori
I'm hesitant, because (1) every scraper I've ever written used a webdriver (usually Chrome) and it ran Javascript and everything you'd expect from a browser, and (2) I'm lazy.
There definitely still are bots using JavaScript, and getting by Caddy, since we do have the bots that are getting through Anubis and are obviously poking for vulnerabilities.
We had a litany of different bots before that, not just Claude bot, but I don't think I actually did (or could, from the logs) see if they use JavaScript. It did seem to stop the big LLM scrapers, though.
Viewing a single comment thread. View all comments