1,303 karma · joined July 18, 2017
I'm pretty personally convinced that it was the supplements that helped here.
This article, for example:
https://www.ccjm.org/content/89/3/154
...cites several cases where daily supplementation of 50K IU was required to restore normal D levels, although also a case where that same dose caused toxicity. As one of the other commenters in the thread noted, working with your doctor to establish the right level is probably the right move. If nothing else, they have the capability to test your serum levels to see where you're at.
GPT-5 and GPT-5-codex are significantly cheaper than the o-series full models from OpenAI, but outperform them.
I won't get into whether the improvements we're seeing are marginal or not, but whether or not that's the case, these examples clearly show you can get improved performance with decreasing resource cost as techniques advance.
Same deal there -- the original author was a genius and was the only person who knew how it was set up or how it worked.
Sure! The study focused on experienced devs working in complex codebases they already knew thoroughly. This was and is the worst case for using AI tooling from a cold start, _particularly_ AI tooling as it existed at the time.
There were also only 16 developers involved in the study.
Time has passed since the study and we've had an entirely new class of tool introduced (the agentic CLI a la Claude Code) as well as two subsequent generations of model improvement (Sonnet 3.7 to Sonnet 4 to Sonnet 4.5). Given that the results of the METR study were stated as an eternal, unqualified truth, the fact that tooling and models are much superior now compared to when the study was conducted is worth noting as well.
So if the "dumb guy" take is "just expose the kids, they'll adapt to it", in the absence of hard evidence to the contrary (and maybe even with it) the average doctor is going to _reflexively_ take the opposite position, because that shows that you (or the conventional wisdom) were wrong.
There are exceptions, and they are either the ones that just don't care at all, or they're the best docs you'll ever find.
1. In my experience, Claude Code (I've used several other models and tools, but CC performs the best for me so that's my go-to) can do well with APIs and services that are proprietary as long as there's some sort of documentation for them it can get to (internal, Swagger, etc.), and you ensure that the model has that documentation prominently in context.
2. CC can also do well with brownfield development, but the scope _has_ to be constrained, either to a small standalone program or a defined slice of a larger application where you can draw real boundaries.
The best illustration I've seen of this is in a project that is going through final testing prior to release. The original "application" (I use the term loosely) was a C# DLL used to generate data-driven prescription monitoring program reporting.
It's not ultra-complicated but there's a two step process where you retrieve the report configuration data, then use that data to drive retrieval and assembly of the data elements needed for the final report. Formatting can differ based on state, on data available (reports with no data need special formatting), and on whether you're outputting in the context of transmission or for user review.
The original DLL was written in a very simplistic way, with no testing and no way to exercise the program without invoking it from its link points embedded in our main application. Fixing bugs and testing those fixes were both very painful as for production release we had to test all 50 states on a range of different data conditions, and do so by automating the parent application.
I used Claude Code to refactor this module, add DI and testing, and add a CLI that could easily exercise the logic in all different supported configurations. It took probably $50 worth of tokens (this was before I had a Max account, so it was full price) over the course of a few hours, most of which time I was in other meetings.
The final result did exhibit some classic LLM problems -- some of the tests were overspecific, it restructured without always fully cleaning up the existing functions, and it messed up a couple of paths through the business logic that I needed to debug and fix. But it easily saved me a couple days of wrestling with it myself, as I'm not super strong with C#. Our development teams are fully committed, and if I hadn't used CC for this it wouldn't have gotten done at all. Being able to run this on the side and get a 90% result I could then take to the finish line has real value for us, as the improved testing alone will see an immediate payback with future releases.
This isn't a huge application by any means, but it it's one example of where I've seen real value that is hitting production, and seems representative of a decently large category of line-of-business modules. I don't think there's any reason this wouldn't replicate on similarly-scoped products.
If it seems to work well, it's because it's copying training data. Or it sometimes gets something wrong, so it's unreliable.
If they say it boosts their productivity, they're obviously deluded as to where they're _really_ spending time, or what they were doing was trivial.
If they point to improvements in benchmarks, it's because model vendors are training to the tests, or the benchmarks don't really measure real-world performance.
If the improvements are in complex operations where there aren't benchmarks, their reports are too vague and anecdotal.
The exciting part about this belief system is how little you have to investigate the actual products, and indeed, you can simply rely on a small set of canned responses. You can just entirely dismiss reports of success and progress; that's completely due to the reporter's incompetence and self-delusion.
The scenario 1 setup could complete the application but it had about 3 major and 3 minor implementation problems. Four of those were easily fixed by pointing them out, but two required significant back and forth with the model to resolve.
The scenario 2 setup completed the application and there were four minor issues, all of which were resolved with one or two corrective prompts.
Toy program, single run through, common cases, stochastic parrot, yadda yadda, but the difference was noticeable in this direct comparison and in other work I've done with the model I see a similar improvement.
Take from that what you will.
I made a crude first stab at an approach that at least uses similar steps and structure to compare the effectiveness of AI agents. My approach was used on a small toy problem, but one that was complex enough the agents couldn't one-shot and required error correction.
It was enough to show significant differences, but scaling this to larger projects and multiple runs would be pretty difficult.
https://mattwigdahl.substack.com/p/claude-code-vs-codex-cli-...
So I put the older and newer versions of Claude Code and Codex CLI to the same task of generating a simple React web app from a specification, using a consistent prompt structure, and compared the results.
It's not a quantitative evaluation, but it is an apples-to-apples qualitative comparison that shows clear differences in agent/model performance on an equivalent multistep task using equivalent approaches.
Without a truly adversarial process (like, say, the US IRS), where there's a party that is actively incentivized to _find_ problems rather than to _not get caught overlooking_ problems, it is never going to drive quality beyond the bare minimum plausible deniability level.