3,193 karma · joined April 28, 2014
I personally feel like we've plateaued in raw model intelligence (even regressed, I find sonnet 4.6 to perform better than Opus 5) but we've gifted them new skills that allow them to run for longer and explore the search space more thoroughly, making them more effective at the same level of "intelligence".
Take two smart people and a problem to solve. Give one person the tools, the other person nothing. The one with the best tools wins. At some point its more about abilities than raw intelligence. Watching a "frontier" model fumble with basic syntax is still common, but not if you give it treesitter.
Well, not really. Massive chunks of the search space can be eliminated through clever (but non-optimal) algorithms. The rest of the search space can usually be explored through heuristics. In practice, we can solve gigantic TSP problems "well enough" and "fast enough".
It reminds me of the Midwit meme. Both the low IQ and high IQ folks say "Heuristics are good enough". Only the mid IQ guy cares about NP-hard.
That's not to take away from the research into theoretical limits of computation. Just noting that's a completely different question from the practical concerns of actually solving those problems IRL
So in practice, Lisp uses tokens very efficiently and tends to get things right, fast. The syntax just is not an issue one has to worry about (and your harness should be checking your model on this btw)
I suspect that Clojure which adds a little more syntax - vectors [] and maps {} - is slightly better than other Lisp dialects since the data structure literals provide a stronger signal than parens.
I'll never forget the cycle, after staunch objections to integration/e2e/manual tests and zealously insisting on full unit test coverage, the senior dev deployed changes ... boom. The process failed hard minutes after start because there was a bug in main. Full production outage. A bug so obvious we should have caught it by eye. Clearly, in the weeks developing the feature, he had never bothered to even run the application. Each part was fully polished in isolation but utterly failed when integrated into an application.
The way I've come to think of it: Unit tests are for libraries, a closed world where you can neatly define all inputs and outputs. Integration/E2E tests are for applications, an open world which require systems thinking, simulations not proofs. Know which world you're working in!
So yes, garbage in -> garbage out, but framed in a way that makes it clear what is garbage. The ideas, not the engineer themselves :-)
This suggests we need to be doing more designing and planning; introducing that friction intentionally to make sure bad ideas get culled, viable ideas get refined. Critical thinking becomes the bottleneck; the quality of the idea becomes the deciding factor in success.
It's important to note though, from the perspective of the LLM's objective function, that this is not a failure at all! LLMs are designed to match patterns. You give a jumbled mess of incoherent ideas, it will faithfully reproduce a token stream of incoherent ideas. It's only when the model output is subjected to the real world that it fails.
"Renewable alternatives cover half of 40% of fossil fuel demand" would be more accurate. Still a win, but it's not "overtaking fossil fuels" by any definition.
The myopic focus on carbon is an easy trap to fall into; it's the quantifiable part of the system. In this world, that which gets measured gets managed.
I would hope we'd conserve these ecosystems because of a deeply-held spiritual connection to life. Or at least a realization of the value they provide to humanity by simply existing. But carbon remains the objective metric by which conservation interacts with the financial market.
I love this sentiment. Quality should transcend the tooling used to create it.
The by-hand artisan coder vs the LLM-assisted artisan coder - not that big a difference in practice. The important part is actually giving AF about your work.
Ultimately it's about skin in the game. Do you care about what you're building or not? Show me, don't tell me.
Conserving these ecosystems is less about their daily respiration and more about keeping existing carbon out of the atmosphere. It's a defensive strategy. Not "carbon capture" per se but more "avoid loosing the carbon we've already captured". We're interested in the sink, not the flow.
Of course when the plant dies and decomposes or burns up, it all goes back. Long term, it's almost exactly net zero. The ONLY way to use plants for long term carbon sequestration is to make sure that plant matter gets treated or literally buried so that aerobic microbes and fungi can't break it down.
The fear with these coding agents is that, unsupervised, one will merge a sneaky pro-AI changeset like this without a human ever signing off on it.
But this looks like run-of-the-mill bad human decision making, no singularity yet! This is fine.
We should put software squarely in the "complex system" category, ie "The only way to find out is to build the software system and try to break it". Writing code may be cheap but running the software IRL is still constrained by both computational bounds and the laws of physics. Not to mention social and market dynamics.
If anything this strengthens the overall argument. "The limit is contact with reality".
Code most definitely does NOT have an easily verifiable search space. P!=NP? Halting problem?
It's easy to brush this off and think those are obscure computer science trivia, but these things come up all the time. If we could prove software correct, we wouldn't need to test it.
> Most of what matters doesn't behave like that. The universe is mostly the emergent behavior of complex systems..... In systems like these, no amount of reasoning delivers the answer, because there's no shortcut hiding in the gaps. You have to run the thing
Software is clearly a complex system, with plentiful examples of emergent behavior. To understand the behavior, you HAVE to run the thing, in prod, on the open network, with real users, under real conditions, with real data. You can simulate these in testing but you can never fully reason about them beforehand - they are computationally irreducible.
The difference is huge. An LLM conversation can contain within it a batshit crazy model of the world and still produce a plausible stream of tokens. No skin in the game. If you don't test your ideas against reality, you don't have a model of reality; you have a model of written human thought. It's critical to understand the difference.
Dedicated coders were somewhat necessary to wrangle the tech stack (which itself grew more complex to accommodate more devs, a positive feedback which nearly spun out of control in late 2010s). But now? A single coder with domain knowledge has a shot at wrangling their own tech stack as a part time job, the rest of the time focussed on IRL value. This is the future I want.
Clustering (for all its faults) is the only off-the-shelf technique for seamlessly switching between these two modalities without having to change the underlying data representation. Need more detail? Zoom in. And the zoom level is adaptive so it works with any scale.
There are better aggregation techniques (summing to a hexagonal grid, heatmaps, etc) but they generally require a separate calculation (possibly server side) and then switching to the raw source for the individual point view, either manually or at some hardcoded zoom level. It's not the same experience - it feels like two separate map layers instead of one integrated clustered layer.
This is mostly a matter of what's available in the mapping libraries. You could imagine building an alternative to clustering that calculates a heatmap on the fly when zoomed out, eventually revealing points as you zoom in. But presently this is something you'd have to DIY. For now, clustering is the only thing that works right out of the box.
Writing manifests seems like a trivial thing to focus on. Who operates the k8s cluster in production? Who runs upgrades? Who's on call to monitor the system? Of course if someone else is doing all the work for you, it feels like free lunch!
It's impossible to efficiently manage that much wealth in one brain. Distribute it!
I'd run an essay contest. Every year the best thousand ideas are selected by yours truly and get funded at $100 million each. Even then, it would take a decade to burn through one trillion. It's a tough job, but I accept.
It seems inevitable that we ask for more AI assistance on topics we don't understand. And therefore have the least context to correct. Result: a flood of poor quality information.
In areas we DO understand, we'll either not ask AI at all, or treat its results with a higher degree of skepticism. Result: a lack of high quality information.
Inevitably this means a higher volume of non-expert prompts gets translated into the next generation of internet content. AIs are pumping out more novice-level text and less expert guidance.
The result will be an internet full content written from the perspective of an ignoramus; not addressing any complex issues, staying surface level on every topic. Which will cascade into future models, etc.
Going back to code as bunch of carefully arranged ascii chars feels like a regression.
This isn't necessarily anti-AI. Imagine a tool that could quiz you, provide context for decisions, and make sure you're up to date on your knowledge of the codebase - instead of just writing code for you. IOW an AI-based system could intentionally add the right kind of friction to improve understanding.