Comment by jeswin | Hacker News Reader

Comment by jeswin | Hacker News Reader

Parent Full thread

I've tried Chinese models extensively over the last 1 year I've been working full-time on this project [1]. They're borderline useless. 90%+ of my code is written by Codex, and perhaps <10% by Claude and myself.

> Can you share any example prompts where Chinese models aren't good enough?

Nobody is prompting any more, it's more planning, automation, guardrails and reviews. For example, if your tests report that there's a 40% slow down in a particular type of analysis, you have to review possible causes, and approve a plan involving multiple agents to find solutions that won't break 50k tests. And that solution itself is a small part of the work that gets planned in a day.

Codex and Claude are the only tools that work for me, inspite of testing almost everything out there to save costs.

[1]: https://github.com/orgs/tsoniclang/repositories

You can't use LLMs without prompting them. But, ok, you use codex and it doesnt feel like you prompt them much, but do you have an example commit from codex which you are sure can't be matched by Chinese models?

Hard to explain this without taking up space on an unrelated thread.

But I'll give you a short example. Let's say you have const x = [1, 2] and we want a policy/solver for the type of x (int32 array, int64 array, or floats etc). We could say we'll infer based on the usage of "x", but it's not free. For example, what if it's passed into a generic function add<T>(...), or if there's reassignment or it's aliased, or if it's being used in a loop counter, or a math operation or a combination of these? This is of course doable, but it requires a sweep of a million lines of code to understand the implications on C#, Rust, Mojo targets, and on the current code and tests.

I haven't tried this specific example with OS models, but this is usually the kind of thing that almost immediately falls short and breaks everything. Codex and Claude will do a decent job of analyzing these and giving me better impact analysis. Codex and Claude will also be wrong many times, but the output is often usable.

Add: When I see OS models coming very close to frontier models in coding benchmarks, I just assume based on my experience that they've been benchmaxxed.

Reply on news.ycombinator.com