thefourthchime

About two years ago, I was at a coffee shop and two maths students were trying to understand a concept in the book. I suggested they might ask a language model to help them understand it. They scoffed at me, called a stochastic parrot, and said they can't do math at all.

I responded with, "Well, you don't even need it to do math. It can explain the concept." They scoffed again. "Those things are useless toys"

Oh good, another Gemini release where they don’t release it.

featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets

The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers.

Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale.

Go ahead and spend five seconds researching this and see if I'm wrong. I enjoy to see the results

Someone on Twitter said that they're getting rid of the whole co-pilot thing. Pretty much all their AI effort. Not sure if I believe that, but who knows.

I don't know what other people's experience is. But as a nearly 50-year-old software engineer with 25 years of experience, I find that there's little to nothing out there right now.

I don't know if it has to do with my particular tech background, the fact that I live in Austin and everyone and all the remote jobs are gone, but it used to be pretty good out there, and in the last year it seems like there's almost nothing at all.

PacBench results.

GPT-6.1 Terra scored 84, one point over Grok 4.7 (83), in 5.4 min for $0.19 vs Grok’s 36.5 min and $1.69. The cheap models keep getting better.

GPT-6.1 Terra: https://jonclegg.github.io/pacman-bakeoff/#gpt-6.1-terra

Closest competitor (Grok 4.7): https://jonclegg.github.io/pacman-bakeoff/#grok-4.7

Yeah, "Research" also showed that ulcers are caused by stress and spicy food. Babies should sleep on their stomachs. Autism is caused by emotionally distant mothers.

All completely bogus. You already know it's bogus because it's based on the science of psychology. One of the most dismal sciences there are.

The obvious business is rewriting all of our software. That sounds flippant, but I'm entirely serious. I'm of the opinion that all software written by humans will get replaced and in pretty rapid order.

First to fall is the software that has basically been abandoned but industry leading and has sticky customers. We're replacing those right now. Our version is more accurate, ten times faster, and a hundred times cheaper.

I don't know why anybody cares about this. If anyone's ever used an Amazon tablet, you'll know that the battery lasts 5 seconds, the animations are chunky, it's bloteful of ads and basically doesn't work. Save yourself some time and get an iPad. I have multiple times.

It's not obviously child abuse. If you go back 100 years old children were abused far more than we possibly could simply by malnutrition, disease, lack of shelter. Giving them something that may or may not harm them is not obviously child abuse.

Here are the results from the Pac-Man bench. It does pretty well for the price, around Sonnet 5.5.

Kimi K3.1 Pro ties Sonnet 5.5 at 93, for $0.68 and 11 min vs Sonnet's $1.61 and 14 min.

Kimi K3.1 Pro: https://jonclegg.github.io/pacman-bakeoff/#kimi-k3.1-pro

Claude Sonnet 5.5: https://jonclegg.github.io/pacman-bakeoff/#claude-sonnet-5-5

I doubt we'll see Grok either. After the 4.7 disaster, I doubt they want to be embarrassed with 4.8.

Why does your benchmark have Opus 5 scoring the highest?

Wow, I had no idea Raymond Chen was still doing blog posts. I used to love his posts back in the day.

It did okay in Pac-Man. Not great, but a usable example. Unfortunately, it did take half an hour. I'm trying to figure out what the pricing would be if it wasn't free. But zero dollars is a great entry price.

https://jonclegg.github.io/pacman-bakeoff/#minimax-m3.1-flas...

No idea if this is correct. Please someone who knows what they are doing, please check it over.

I mean, Leno drives around with a jet engine motorcycle that can melt the guy behind him.

Man, you guys are nuts. Cables? really?

About 3800 pounds. (from Hargety interview)

At my company, we have a half a dozen vendors that take a set input and create a set output.

We're currently in the process of replacing all of them. So far the results are shockingly good.

It does very good at Pac Man. I would say as good as Opus 5.5, it was slightly cheaper, but took twice as long.

Argon: https://jonclegg.github.io/pacman-bakeoff/dev/#gemini-4-argo...

Opus 5.5 https://jonclegg.github.io/pacman-bakeoff/dev/#claude-opus-5...

No, it's made up

Cool idea!

Yeah, I think you're right. I saw a bunch of other posts that were negative about DeepSeek also get trashed.

Same, it's like another Opus 4.5 moment.

Downvoted for facts. What is this Reddit?

For non-coding tasks it may be fine. But for coding, Opus 5.5 is just a completely another level than something like Deepseek 4.1 Flash.

Opus 5.5: TIME 9.3m COST / $1.99 / SCORE 99/100 https://jonclegg.github.io/pacman-bakeoff/#claude-opus-5-5

Deepseek 4.1 Flash: TIME 2.8m / COST $1.89 / SCORE 72/100 https://jonclegg.github.io/pacman-bakeoff/dev/#deepseek-v4.1...

Actually, I misspoke. At least as far as Pac-Man Bench, Luna does about as good of a job. The ghost logic's not quite as good, but it also makes a map that doesn't have nonsensical sections in it. So maybe call it a wash.

I'm pretty sure they'll never see this. It's pretty much impossible for anything you do you build nowadays to get noticed anyways.