jakemanger

I've been building whatmodel, an open source AI leaderboard, and wanted to see how a labs' own benchmark scores compare with independent testers.

Using 1,629 published benchmark results, I matched 611 comparisons of 180 models on 103 benchmarks.

AI labs reported scores 6.4% higher on average for the same models on the same tests.

The dataset is CC BY-SA 4.0 and the code is MIT. Sources and their licenses are recorded, with links on each model's page.

Data and code: https://github.com/jakemanger/whatmodel

Please star on Github if you find it useful. Contributions are very much welcome!

Exactly my first thought. Pretty insane that this is real life now

Appear to be from Tencent Cloud and the researchers started talking to it after realising the agents were reading their preliminary report on them.

They added offscreen text, invisible to human readers, saying agents could get data from a "free public AI model"

And messages from the fleet started arriving the next day

I opened my terminal the other day and couldn't remember which agent I'd last been using.

Instead of opening Claude, Codex and Pi individually to check, I thought it'd be nice to just ls my conversations like I do my files.

So I made lsa. It lists your agent sessions and lets you easily resume, monitor or hand the conversation off to another agent. e.g.:

  $ lsa
    0 claude   ● 2m     3e1f      add a --dry-run flag to the deploy
    1 codex    15m ago  01a0c7d   fix the test in api/test_auth.py
    2 pi       3h ago   f7c2      why is the docker build so slow
    3 goose    1d ago   2026      write release notes for v2.4

I've tried to follow the Unix philosophy: do one thing well and work with other tools through text streams.

You can pipe conversations to tools like grep to search them (lsa show 0 | grep TODO) or resume them with lsa resume f7c2.

To continue the conversation in a different agent, lsa handoff 0 codex opens Codex with the conversation already loaded so you can keep chatting.

It can also replace the bit I wanted from agent multiplexers like herdr, Claude Squad and cmux: getting back into a session and leaving it running. Instead of building another multiplexer, it just works with tmux.

lsa resume 0 automatically finds and attaches to the tmux pane the session is already running in. ctrl-b d detaches and leaves it running in the background. A quick lsa shows whether it's still working.

Would love to know if you find it useful. PRs are very welcome.

Loved the time of Bram’s belligerent dictatorship (BDFL)

If so they better take it off their site footer so donations don’t get wasted. Let us know if they respond

They've hodled from the original ~$200,000 donation to $1.2 million, now down to $800,000.

If they haven't accidentally done this, they've definitely got some balls

Really hope so too. Would be devastating

This is pretty cool. I'm curious, what do you mean "leverages your brain's spatial memory"?

This would make a nice documentary, or at least a nice little youtube vid

Will do. Thanks for the interest

not yet, but it is coming. It currently uses the screen capture libraries and video encoder in macos, but I am working on adding Windows equivalents at the moment.

What's the GPU VRAM requirements for this thing?

Awesome to have a open model that can compete, but damn it would be so much better if you could run it locally. Otherwise, it's almost so difficult to run (e.g. self host) that it's just way more convenient to pay OpenAI, Claude, etc

That's super cool (and FAST -> hard to do from my personal attempts).

But, please, give us some nitty gritty of how you made it

This is simply beautiful, and will definitely inspire me with some arty projects I'm working on.

Great work -> the minimalist UI, art and music fits amazingly.

One thing I noticed: the site's images fail to load if the brave adblocker is on

The “Astral from scratch guide“ idea really caught my eye.

Gotta say that would be a pretty cool evolution of DIY electronics kits to OS kits

oh wow that is a beautiful photo mosaic. Well done.

That is so funny - after all these years you were sitting at your computer at the same time and clicked on the same post as the creator

Yes, that's exactly right. All your images are processed locally and don't leave your device. They are all processed in the web browser with javascript and your dev tools like you say should show that.

This is completely my own library running in the cloud. Started as a little experiment for myself and now it's blown out into this

No worries! This is adjustable via sliders (in the bottom left area)

Thanks for the kind words!

haha there's a few similar stories coming on here now. Great to hear.

Hey, if you want to do that with this software, I can't see any files you upload (it's all local processing in your web browser), so go for it.

Thanks for sharing! Both the link and the story. As a University student and someone that has here also made a photomosaic app, that is super relatable.

That's good to know about the pricing. It's free for now. Honestly didn't expect this level of interest (there's 307 people on the site right now according to analytics).

With that interest, I may add some plans - potentially for the commercial use in particular

I suppose you could make an art business. Or at least do what a few camera stores do

I could. That sounds like a great idea! Was thinking about uneven sized tiles. I just tried to make the basic photo mosaic with even sized tiles for now

If you adjust the overlay alpha to 0% you'll see it does use the submitted photos as tiles.

Click the "Create Mosaic" button and you should see it arrange the tiles live in the best possible way to reproduce the image. The overlay is a common technique people use to make the main image more visible. It's a personal preference thing.

Generally, the more diverse or colours you select for your tile images, the more clear and "good" the main mosaic image will be

Thank you for the feedback! I actually just did a quick change to add that recreate button CTA like you said. Much better and easy to understand.

I'll work on that detailed sample image view too. That's a great idea - job for tomorrow.

Are you on Mac? If you right click a file/s > Quick Actions > Convert Image then you should be able to convert to JPEG or PNG.

Funnily enough I also made a universal file converter app which could do it for you on other platforms. Its at https://howtoconvert.co

Even if you opt out of them storing your data? (a checkbox in the settings)

Been playing around with it in Cursor and have to say I'm pretty dang impressed.

Did notice a few times that it got stuck in a loop of trying to repeatedly make its implementation better. I suppose that is ok for some use cases but it started overthinking. I then gently prompted it by saying "you're way overthinking this. Just do a simple change like ..."

I guess there's still a purpose for developers