17,811 karma · joined June 4, 2016
With AI, the correct approach is to think more like a software architect.
Learning to plan things out in your head upfront without to figure things out while coding requires a mindset shift, but is important to work effectively with the new tools.
To some this comes naturally, for others it is very hard.
That said ... I do think Codex 5.2 was the best coding model for more complex tasks, albeit quite slow.
So very much looking forward to trying out 5.3.
From the post I expected that the tasks were about analysing traces, but all the tasks in the repository are about adding instrumentation to code!
Some of the instructions don't give any guidance how to do it, some specify which libraries to use.
"Use standard OTEL patterns" ... that's about as useful as saying "go write some code". There are a lot of ways to do instrumentation....
I'd be very curious HOW exactly the models fail.
Are the test sets just incredibly specific about what output they except, and you get a lot of failures because of tiny subtle mismatches? Or do they just get the instrumentation categorically wrong?
Also important: do the models have access to a web search tool to read the library docs? Otel libraries are often complicated to use... without reading latest docs or source code this would be quite tricky.
Some models have gotten better at adding dependencies, installing them and then reading the code from the respective directory where dependencies get stored, but many don't do well with this.
All in all, I'm very skeptical that this is very useful as a benchmark as is.
I'd be much more interested in tasks like:
Here are trace/log outputs , here is the source code, find and fix the bug.
So a 100bn deficit out of 800bn total US imports.
The deficit is there, but it's not nearly as lopsided as some reporting would have you believe.
Massive endeavor for a lot of setups.
The existing AI plugins for neovim aren't great.
And their logo is, lo and behold ... a hand!
Most popular programming languages are optimized for human convenience, not for correctness! Even most of the popular typed languages (Java/Kotlin/Go/...) have a wide surface area for misuse that is not caught at compile time.
Case in point: In my experience, LLMs produce correct code way more regularly for Rust than for Js/Ts/Python/... . Rust has a very strict type system. Both the standard library and the whole library ecosystem lean towards strict APIs that enforce correctness, prevent invalid operations, and push towards handling or at least propagating errors.
The AIs will often write code that won't compile initially, but after a few iterations with the compiler the result is often correct. Strong typing also makes it much easier to validate the output when reviewing.
With AIs being able to do more and more of the implementation, the "feel-good" factor of languages will become much less relevant. Iteration speed is not so important when parallel AI agents do the "grunt work". I'd much rather wait 10 minutes for solid output rather than 2 minutes for something fragile.
We can finally move the industry away from wild-west languages like Python/JS and towards more rigorous standards.
Rust is probably the sweet spot at the moment, thanks to it being semi-popular with a reasonably active ecosystem, sadly I don't think the right language exists at the moment.
What we really want is a language with a very strict, comprehensive type system with dependent types, maybe linear types, structured concurrency, and a built-in formal proof system.
Something like ADA/Spark, but more modern.
Of course dynamic dispatch can be implemented in almost every language. The Linux kernel uses dynamic dispatch with C!
But that's a hack, not a language feature.
Or, even simpler, just turn off over-commit.
But if swap comes into the mix, or just if the OS decides it needs the memory later for something critical, you can still get killed.
In the past I used it to try out Swift, J, R, ...
The difference when working on larger tasks that require reasoning is night and day.
In theory it would be very interesting to go back and retry the 2024 tasks, but those will likely have ended up in the training data by now...
There's a crazy amount of complexity and customizability in systems like ERPs for multinational corporations (SAP, Oracle).
When you start with a small town, you'll have to throw most of everything away when moving to a different scale.
That's true for software systems in general. If major requirements are bolted on after the fact, instead of designed into the system from the beginning, you usually end up with an unmaintainable mess.
Treat it as a developer that just joined the project and isn't aware of the conventions.
Provide hints for the desired API design, mention relevant code locations that should be read to gain context on the problem, or that do similar things.
An AGENTS.md that explains the project and provides some general guidelines also helps a lot.
Codex can be incredibly strong when prompted the right way.
* Epidemic broadcast trees: https://asc.di.fct.unl.pt/~jleitao/pdf/srds07-leitao.pdf
* HyParView: https://asc.di.fct.unl.pt/~jleitao/pdf/dsn07-leitao.pdf
The iroh-gossip implementation is based on those: https://docs.rs/iroh-gossip/latest/iroh_gossip/
The old one for dedicated servers (robot) is horribly outdated though.
The whole Rust ecosystem is heavily biased towards prioritising memory safety and "safe by construction" .
This is evident in the standard library, in how crates approach API design, what the compilation defaults are, ...
In 6+ years of using Rust the only time I had to deal with segfaults was when working on low level wrappers around C code or JIT compilation.
Zig has some very interesting features, but the way they approach language and API design leaves a lot of surface area that makes mistakes easy.
It might take some people months rather than days, but I think that is a desirable outcome.
Important low level software should be written by competent developers willing to invest the effort.
Azure is ... a different story...
I indeed was not enabling it.
I'll give it a try!
(dioxus-7-rc.3)
It usually only works when reordering elements, or changing static values (styles, attributes, etc).
Which, to be fair, does speed things up a lot when tinkering with small details.
But 70%+ or so of my changes still result in recompiles.
So many UI frameworks don't have these and require building them yourself...