robotmay

I couldn't even stand 4.6 by the end. 4.5 was very usable, 4.6 started alright and somehow got more annoying. 4.7 made me quit my subscription and ditch their services entirely - it's an honourary member of my quite short "Coworkers I'd like to throttle if I didn't work remotely" list. Infuriating to instruct or communicate with.

It's very good at Elixir in my experience too. And it just does what I ask and doesn't wind me up like Opus. I don't think I've had to insult it more than once per day.

Unrelated, but this week I've been on a memory binge with the Wayback Machine, trying to find old content of mine from the early 2000s. Took me a while but I've finally put together a good bit of info about myself at the time that I'd completely forgotten, and it's all thanks to the Internet Archive storing my little gaming review website from when I was 16. I could barely remember any of the other stuff, it's been genuinely surprising figuring out what I'd forgotten. I couldn't even remember most domains I owned aside from one, which I used as the starting point.

Still can't remember what my Tripod site address was, but that might be lost to time.

Thank you, Archive.org.

Bit too early for me unfortunately, I was 6/7 then! I was still having a lot of fun up until about 2005, before I went off to university. The past week I've been trawling the Internet Archive to try and find things I vaguely remember - it's remarkably hard. So much of that era is missing, though thankfully one of my own sites was indexed and I'm able to fan out a bit from there. Albeit mostly to French anime fansites, but I'll take it.

I'm with you on the YouTube side - I stopped making my own videos earlier this year when I burnt out, and now I'm finding myself increasingly irritated just by the thumbnails when I visit the site. I'm definitely viewing less than I did last year.

I hadn't considered the impact on manga. Machine translations are probably helping with fan translations for more niche projects. I haven't seen any obvious AI art myself yet but I do tend to read stuff from smaller creators on slower release cycles.

Honestly the golden era ended around 2010, once the normies showed up en-mass for social media. The late 90s and early 2000s were so much fun on the internet, and I'll always be sad that today's nerdy teenagers won't get to experience that.

I have a few ideas for my own retro internet projects but I'm making sure I can protect them from the AI scourge first, or at least do my best to.

Haha I did debate going much worse with the junk images but wanted to err on the side of caution in case I subject possible clients to something like goatse.

I did something like this a few weeks ago on my photography site: https://robertmay.photography/journal/meta-has-tried-to-scra...

Meta not only hasn't noticed, but is currently sending about 11 requests per second to my site. I've also seemingly trapped one of those TV proxy scraper nets as I'm getting absolutely hammered by requests from all over the place now. I get maybe 10 legit visitors per day, and I'm currently blocking 406,787 IPs from things that have fallen into my honeypot.

I've tweaked my site to return empty status responses a configurable amount of time but the traffic has been so intense that Traefik is now struggling, so I'm going to have to figure out something else. I was returning over-capacity errors and I think that was a mistake, I've swapped to 400 range status codes now. I don't want to use Cloudflare so I'm not sure what to do after this.

The people at these companies are either incompetent or malicious.

The photo they've chosen for the article even supports this. It's like he's leaning on a bar twirling his moustache whilst giving you the side-eye.

Just on the off-chance anyone should see this: if you were one of the brothers involved in running the Blinkys "R" Us forum back in the early 2000s - I still think of it regularly. I can't even remember what my username was these days, and Archive.org never archived it at the time, but I still remember it very fondly.

Now that’s a name I’ve not heard in a long time.

Yeah I've been really surprised by how inept they are. Lots of others have fallen for the trap but extract themselves fairly quickly and probably blacklist my site. Archive.org got stuck for a while, which was unfortunate, so I had to exclude them manually. There's something on an Oracle network that keeps coming back but it's very slow compared to the ridiculous rate of requests from Meta's scraper.

It's up to 1.5 million requests now, sigh. My guess is that Meta has too much money and is in the "move fast and break things" stage where they're just throwing money at a problem. Not setting a max crawl depth is hilarious though.

I've swapped over to it in the past two weeks, it's been really good. It does what I ask and doesn't think it knows better than me, which so far has made it the most pleasing experience I've had when slop-coding.

My only wish is it were somewhat cheaper, as it tends to balloon pretty quickly when I'm using it in Opencode. I'm currently trying to offload a lot of work to subagents to stop the context expanding so rapidly. But on the upside, I rarely have to correct it - I've spent far less time arguing with this than with anything else so far.

Yeah Meta's bots are well and truly stuck in the blackhole on this site at the moment. I have no idea what they're supposed to be doing, but in case they are using them for AI work I've just added a token multipler based on subnet totals so they can have some more fun. I might come up with some other fun response types for them too.

In an odd sequence of events, I've sorta got a weird tiny following in China due to having a vintage camera stall in an antiques market here in Wales that gets shared on Xiaohongshu (https://en.wikipedia.org/wiki/Xiaohongshu) sometimes when I have Chinese customers. So I'd feel bad blocking the entire country myself, though thankfully I'm not being inundated with traffic from there yet - the most egregious bots I've caught so far are actually ones from Meta which ignore robots.txt.

If I get some time to un-LLM-ify it then sure! I build all my sites in Phoenix these days anyway so I'll likely want to port it to my other sites at some point too.

I do want to add some more gibberish generation to mine. Once a visitor hits the honeypot route it does replace the rest of the site content with nonsense but I think it'd be fun to have it spewing really daft, really long LLM-generated essays on toasters.

Currently not really an issue on this site but connections aren't an issue on Elixir usually anyway, unless you get up to about 1 million on one machine IIRC.

It's mostly about luring them into URLs they're explicitly told in robots.txt that they shouldn't index. I do some identification via reverse DNS of known crawlers I actually want like Googlebot, though they respect robots.txt, in case something goes wrong and they accidentally get flagged.

Yep this is how I'm doing it. It's marked as my super exciting client archive full of high-res RAW photos that all bots are told not to index in robots.txt, only for some weird reason the link never gets rendered to the page :>

I also explicitly tell all crawlers not to index the images on my site, actually, so I feel even better about serving the ones that do complete junk.

Not sure why I get more crawler traffic than I do legitimate traffic tbh - it's a relatively new website that I've been trying to sort out the SEO for as I'm trying to get photography work, so it's probably started cropping up on the radar of scrapers looking for training data.

I've added a few screenshots above!

Elixir's really fun to do this in because the BEAM will let you have hundreds of thousands of processes sat around doing nothing quite happily, so slow IO and such is something it's weirdly well-suited to.

This is on my photography site so I'm less fussed about them harvesting my writing, and more about causing problems for image harvesters. I'm sure they'll get some stuff anyway but at least one bot got stuck in a 400 page hellhole earlier today so I'm overall very pleased :D

Thankfully this is my photography website so what they're mostly after is images, which they have to request. I'm also intercepting all my legit images once the bot is flagged, which will no doubt have unintended consequences but I'll find out at some point I guess.

Haha here's a few snippets (will auto expire in a week, future readers):

https://ibb.co/mCgXPZPg https://ibb.co/qFxVRQGW https://ibb.co/9m6xV1xg

I only just added the token wasting measure so I'm really curious to see what happens if another bot loads over 400 pages.

Gemini's surprisingly bad at it but so far I have some voluptuous curvy toasters, toasters in the shape of a butt, and robot girls with toasters for tits. I have a Gemini subscription for the month after mistakenly thinking I'd get cheap Opencode usage through it, so gotta use it for something.

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it.

Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources.

Currently I trick bad scrapers into a fake infinite black hole path with the promise of tasty data, then serve images to them one byte at a time over 15 minutes (after sending the header quickly), bloat the responses to cost them tokens, and randomly return AI generated images of sexy toasters. I have an admin dashboard with a little leaderboard for which ones get the most stuffed, and it keeps my heart warm on these wet autumn evenings.

If it's anything like the PS4, you might be able to fix this by running a HTTP proxy using Squid and configuring that on the PlayStation. There's something really funky about Sony's networking on them, and configuring a squid proxy for my PS4 vastly sped up my downloads.

I tried to find an article covering this but I've had no luck, so I'm just going off memory here, but I'm pretty certain that's all that was required.

Sigh, frustrating for me but I guess I understand the rationale, if not the implementation.

I'm retired from programming professionally and I liked using Sourcehut for my personal projects because it just got out of my way, and I could put my mostly-old open source projects up there without worrying about people trying to contribute to them thanks to the friction of the system. I do slop-code hobby projects because I'm pretty much done with programming for fun now, but they're always private and I know that my relative load is low as I don't do much different from how I would personally use it.

Personally I'm in favour of blocking LLM pull requests/merge requests/patches to open source repos - the sheer volume of crap is out of this world. I saw a repo with 1.1k unmerged PRs yesterday! The code can be from an LLM but the request to merge it damn well better be written by a human. I'm not going to put effort into reviewing something someone else couldn't be bothered to even write a blurb for.

But for private projects, just shove a rate limiter on the endpoints and be done with it.

Lately I'm very conflicted about software development as a whole. I've always aspired to be good, both in my work and in my relationship with others in the industry. But I can't be arsed any more - the joy has gone out of software, and I just want it to be done and out of my way. And now I have to move my photography portfolio repo that has fewer commits per day than I could write by hand, because there can't be grey areas or compromise on the modern internet.

I used to work for GitLab, for what it's worth (and they've gone quite far down the AI toilet), and I honestly don't think I can be arsed setting up that or anything else. I might as well just shove bare repos on my NAS and go do something I actually enjoy.

Yeah I test drove a Honda E. I don't know why but they somehow used a webcam from a 1999 laptop for the cameras. It was a shame as the interior of that car is gorgeous, but the range is absolutely appalling and those mirrors are awful.

Ooh I’ve been thinking about this idea for years, I’m glad someone beat me to it. Guess I have something to play with over the rest of the weekend!

Ecosia and Qwant are the two EU-based ones I know of currently, though I'm sure there are more.

I similarly stopped subscribing when I realised they were paying money to Musk for grok, and I refuse to fund that man in any way. I hadn't seen the Russian data usage. The Kagi founder above states that using these services doesn't imply support of the governments or owners behind them, but that's a crock of shit. Using their services is, at best, an indicator of a complete lack of principles to a degree that I can now never use their service. He asks for respect for his moral line above, but no-one owes him that - not every viewpoint deserves respect.

Being principled in your choice of services as an individual these days is challenging, as so many companies pursue money over all else. It is doable, however, and Kagi won't feature in my life again no matter how many cool features they add. The internet has become such a cesspit that it warms my heart to know there are still other people who believe in being good and principled, not just making money.