A recap of the Jstpst updates since June

Submitted by twovestsadmin in meta

(I wrote these bullet points. Not a robot. This is just how I write informative posts.)

I wrote a ton of posts in the past few days about the updates. Here's the Official Details and a timeline of what happened:

  • Aggressive bots get SUPER aggressive: Since around June 3rd, we saw a huge uptick in aggressive scrapers, slowing down the site and pinging the CPU at 100%. Not good! It made JstPst very slow.

  • Borked but fixed: Around then, I borked the site in an unrelated upgrade, but then fixed it. 😎

  • Robots.txt added, but not enough: Due to a configuration issue, our robots.txt (see here) was inoperable for most of the past year. Traditionally, this is a small file meant to guide crawlers, and to ask them not to crawl sites. It's an honors-system approach that was actually well respected, up until the explosion of AI garbage post-2020. After fixing this, we saw fewer scrapers, but the most aggressive ones did not give a shit.

  • Anubis to the rescue: Did you see that anime girl appear before you loaded JstPst? That's Anubis. It's a small tool that has helped to totally block the most aggressive bots.

    • Not cryptocoin. Worry not for the "hashes", we're not mining cryptocoins on your computer. The cryptographic challenge is structurally similar, but the point is to have each machine use cycles to get access to the site, to make it economically unviable for bots.

    • Not Cloudflare, not a service: Anubis is self-hosted on JstPst's servers, just likst Postmill. We're not using Cloudflare or another external service. The privacy of your detals have not changed.

    • Hammers: The JstPst Terms of Service now has a clause. People who scrape our site agree to the "We Can Beat You To Death With Hammers If You Scrap Our Site" Clause. The clause doesn't limit it to just Hammers though. The clause is also a joke.

    • Some logging, for now. We have some logging enabled while we respond to the robots. There are still bots in the logs, and they still account for most of our traffic. There's a fucking evil one from the IP "66.249.79.171", from some guy named "Google", and so our hammer lawyers are getting ready.

But I want to hear from you. Anubis has two huge, HUGE problems that I don't like:

  • JstPst now requires Javascript and cookies; is this a problem for anyone? The Anubis gateway requires Javascript (to crunch the Anubis challenge) and cookies (to store the result of the challenge, so you don't need to crunch it on every click).

  • JstPst now requires more CPU power; is this a problem for anyone? Postmill is awesome for being something that you could browse with a budget laptop from 2005. I know this because I used to! It's a very performant site. I no longer have that laptop. But now, Anubis has to crunch some computer numbers. It takes me ~150ms on average on a nice CPU. We can moderate this "difficulty level". Someone with a much slower CPU might see it take longer.

TLDR: JstPst uses Anubis now. Please enable Javascript and Cookies if you don't already. And please please please let us know if this is causing an accessibility problem-- it's not worth kicking out bots if we're kicking out you too.

11

Comments

You must log in or register to comment.

hollyhoppet wrote

i tried to go "no js" for a while but honestly it felt like such a blunt tool for so complex a problem. no shade on anyone who wants that but with modern ui expectations it's really hard to go back to that world

8

cowloom wrote

I'm a no-js user, and it's definitely annoying. Over half of all websites are non-functional without JavaScript, and many that do work have major formatting issues. But I have to do it for my privacy's sake, so I've just accepted that my internet experience will be severely limited forever. This is the only website that isn't read-only for me.

6

flabberghaster wrote

Yeah I feel that the fact we need Turing complete script engines just to read posts is a damn shame, but at this point the battle is lost. Which is sad. But given that no js is so rare and without the JS we're just getting hammered by the Bot Plague it's a small sacrifice to make to keep this place running smoothly.

5

emma wrote

regarding the cpu power thing, anubis completes in 10 seconds on my 15 year old netbook with a single core atom cpu, 2 gigs of ram, and windows 7. this thing takes like 30 seconds to load a modern social media site, so i doubt anyone would be using a device slower than this as their daily driver

Sent from my ASUS® Eee PC

7

oolong wrote

i'm away from from old as balls laptop for two weeks but i'll check when i get back

6

cowloom wrote

I had to enable javascript briefly to get past the Anubis challenge. I suppose it's not a huge issue if the code is all self hosted; the main reason I disable it is for privacy and anti-fingerprinting purposes. The website seems to work if you immediately disable javascript after passing the challenge. Still, here's hoping they come up with a no-js solution.

5