Comment by adriand | Hacker News Reader

Comment by adriand | Hacker News Reader

Fascinating video. I watched almost the whole thing without planning to, I got sucked in.

This is one of those examples of software that reminds me of my struggle to understand how LLMs are passing code evaluations that culminate with people declaring that they are now better than even the best human coders. I have tried to get LLMs (specifically, Claude and ChatGPT, trying various models) to assist with niche problems and it's been a terrible experience. Fantastic with CRUD or common algorithms, terrible when it's something novel or unusual.

The author creates his own version of a "FLIP simulation". I'm going to go out on a limb and posit that even ChatGPT's unreleased o3 model would not be up to the task of writing the software that powers this pendant. Is this incorrect? I realize perhaps that my comment is a little off-topic given that this is not an AI project. However, this project seems like an excellent example of the sort of thing that I am quite skeptical the supposedly "world-class" artificial software engineers could pull off.

I've implemented fluid mechanics using Claude (through Cursor) and it had no problem writing the logic and integrating it using my custom physics engine and custom renderer.

So no, I don't think your assessment is correct. LLMs shine when they get to implement something from scratch on a blank slate with clear API boundaries, whether it's a CRUD app or a physics simulation. Where I think they struggle the most is in big legacy codebases on tasks spanning multiple modules with lots of red herrings.

I'm a computational fluid dynamics (CFD) engineer. Like others have said, I'm pretty confident basic CFD algorithms are in the training data of many LLMs. I would say a bigger problem is the accuracy of the generated simulator. A LLM would not be able to generate good tests. You need both tests for the math ("verification") and tests for the physics ("validation"), and LLMs can't do either at the moment.

Gold-standard verification tests are constructed using the "method of manufactured solutions" (MMS), which can be largely automated with computer algebra software, but are still quite tedious. I know from experience. I don't believe LLMs can handle the algebraic manipulation here particularly well.

Worse, LLMs won't be able to produce actual experimental data to do the validation test with. You'll need to track down one or more experiments from the literature or do your own experiment. LLMs might in the future be able to point you to appropriate experiments in the literature, but they don't seem able to do that at present. I think LLMs might provide useful advice when a simulation ends up not matching the experimental data. LLMs seem to know a thing or two about turbulence modeling, though I would question their knowledge of the most recent advances.

(If you're only interested in fluid simulation for games or computer graphics then physical accuracy is not a priority. But you probably should still use MMS to make sure you've implemented the math correctly. MMS is an interesting technique that has no parallel in software testing in general. Abstractly, the idea is to make a minimal modification to the software so that you have an oracle, and the nature of the modification is such that the modified software passing the test implies that the unmodified software would also pass the test. This idea probably can be applied in other areas.)

You could go and try it out. You'd be surprised how capable the LLMs, specifically Claude are now at creative projects such as this

The "issue" with saying an LLM can't do this is that CFD simulations are not actually that niche. Many university courses ask their students to write these types of algorithms for their course project. All this knowledge is present freely on the internet (as is evident by the Youtube videos that the author mentioned), and as such can be learned by an LLM. The article is of course still very impressive.

Great point. Niche to me, but not to thee. I was unaware. This is actually one of the frustrating things about the LLMs - they don’t tell you when what you asked for is outside their training data!

Claude is pretty good at writing Python programs.

Reply on news.ycombinator.com