14,247 karma · joined September 2, 2012
Now: benchmarking AI at: https://quesma.com/blog/
Previously: co-founder & CTO at https://quantumflytrap.com/
Why do we expect that for other professions it works differently?
How Opus 5.5 imagines these cities is a question for Opus. The real experience is how we imagine - and I believe there are no two people who have exactly the same mental image of an invisible city.
AI, however powerful, is a tool, only as important as the amount it helps mathematicians. Creating mathematics without human understanding is as sound as mass producing copies of Michelangelo David.
“Mathematics, rightly viewed, possesses not only truth, but supreme beauty — a beauty cold and austere, like that of sculpture [...] yet sublimely pure, and capable of a stern perfection such as only the greatest art can show.” - Bertrand Russell, "From The Study of Mathematics" (1902)
“A mathematician, like a painter or a poet, is a maker of patterns. [...] The mathematician's patterns, like the painter's or the poet's, must be beautiful; the ideas, like the colours or the words, must fit together in a harmonious way. Beauty is the first test: there is no permanent place in the world for ugly mathematics.” - G. H. Hardy, "A Mathematician's Apology" (1940)
Source code: https://github.com/stared/invisible-cities-opus-5.5
And a write-up: https://80.lv/articles/artist-turns-bedroom-into-interactive...
- Brainstorming with AI is fine, but it takes skill to use it as food for thought, rather than as a source of answers.
- Do NOT structure your blog post with AI. Once the structure is there, it is almost impossible to humanize it.
- Do NOT use it for writing a draft. Same as above. A few times I did it, and ended up rewriting from scratch. Unsloppification is hard.
- AI for proofreading is wonderful. But I always prompt with "only fix grammar mistakes, do not change the style" and actually look at all changes. If not kept in check, it too eagerly homogenizes style.
If you would like to do so, it is easy (and orders of magnitude cheaper) than running benchmarks.
On wikitext2 there was no difference. I concluded these have no long-term dependency and I should use Linux kernel. Still, the same.
So yes, KLD depends on the dataset. Still, it does not measure what any e2e test does.
I mean, there so many headers and layers, it is tricky to make a choice that will resonate with our intuition . Is it some weighted average? Or maybe ablation test?
It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a Bernoulli distribution. But more likely, many problems are easy (i.e. each run will solve them consistently), many are too hard (i.e. no run is going to solve them) and only a fraction is somehow in between.
Maybe there is some good trick to find a proper distribution, but to my knowledge, we would need to run it at least two times on TB2.1 to get any more educated estimates. That said, I am open to new ideas.
That said, I consider frequentist probability a dirty trick, and that Bayesian is the proper way of doing things (vide David J.C. MacKay" Information Theory, Inference, and Learning Algorithms" and Cam Davidson-Pilon "Probabilistic Programming & Bayesian Methods for Hackers" https://www.inference.org.uk/itprnn/book.pdf, https://dataorigami.net/Probabilistic-Programming-and-Bayesi...).
So, if there are irrelevant remarks, these are mine. :)
Charts are vibe-coded - but it took quite a bit of hand-holding to get something decent. And the logarithmic scale for model size is my conscious choice (against Claude's initial ideas).
You will also see hos long it takes - and what is thd difference between an idea and making it real.
Maybe it suffices to make this pre-title "the ai-design-slop fingerprint · defs 2026.09" lowercase.
In any case, from what I see what is AI aesthetics on the website: pre-title and horizontal lines.
It contrasts with times before, when we actually had to wait for a monthly magazine, and even if we wanted to watch a movie, it was aired (say) the next Tue 7PM.
While now we have all convenience, there is no anticipation - or rest.
Compare and contrast with ARC-AGI, BabaIsBench (https://quesma.com/benchmarks/babaisbench/), or MazeBench (https://mazebench.com/blog?post=introducing-mazebench).
In particular, in one Baba Is Bench post (https://quesma.com/blog/baba-is-aug-2026/), while quoting a Pareto frontier chart from AA, I noted:
> Intelligence Index vs. Cost per Intelligence Index Task from Artificial Analysis. Note that it is based on score of benchmarks like Terminal-Bench v2.1, SciCode, Humanity’s Last Exam, GPQA Diamond - not necessarily fluid intelligence like in abstract puzzle games of ARC-AGI-3 or Baba is You.
* how does quantization hurts pelican-drawing skills (https://quesma.com/blog/qwen-quantization-quality/)
* how does it hurt knowledge (https://quesma.com/blog/quantization-hurts-knowledge/)
But as from seeing various runs, errors bars are gross overestimation (as not "the same test", but "if we have different tasks from the same sample").