XCSme

It's around Qwen-3.8, and Sonnet 5.5 level, but a lot cheaper. It is also really fast.

My tests for Haiku 5.5: https://aibenchy.com/compare/anthropic-claude-haiku-5-5-xhig...

Twice as expensive as Luna, but also considerably smarter too:

https://aibenchy.com/compare/anthropic-claude-haiku-5-5-xhig...

You can assume that all models above a score of 9 are "good", then it's more a matter of preference for their response times, costs, personality and strength in specific areas.

Opus 5.5 are best at coding, and even art generation, but struggle a lot with built-in knowledge, when web search is not available.

Yeah, it is saturated.

It's actually quite hard to make a single task/question with few steps max that SOTA models fail on. Gemini are on top, because asking domain specific questions still breaks most top models, but not Gemini ones.

Claude is lower, and also most Antrophic models, because (historically) they have one of the highest refusal rates and also struggled more with precisely following instructions (i.e. outputting in a specific format/way).

My own tests/benchmarks.

There is no model passing all the tests. Gemini is on top because questions also include general knowledge, trivia or domain-specific questions, which Gemini models are way above any other model.

Even if models have similar scores at the top, it's useful to compare costs/response times.

I am not sure most people will be able to run this locally, and compared to GPT-6 luna it is much slower, verbose and also performse worse.

GPT-6 Luna vs Mistral Large 4: https://aibenchy.com/compare/mistralai-mistral-large-4-0-hig...

I tried testing it, but reasoning effort indeed seems to be broken somehow.

I tried testing it, but reasoning effort parameter doesn't seem to be there and output is sort of broken because it reasons directly in the output tokens...

Yeah, this was known since 3.8 27b was released, the high version reasons way too much, context gets polluted, so answers get worse, assuming it has to do a deterministic task, not a creative one.

Check other benchmark sites/videos too, they all said Qwen 3.8 27b is better on low than high.

Both models on high are kinda bugged and don't give better results. Use them on low, unless you are generating images/art/design.

The exact quant is mentioned on model page[0] IQ3_S, and I think 27b was Q4, via Ollama, the one that fits on a 3090 24GB

[0]: https://aibenchy.com/model/qwen-qwen3-8-flash-next-low/

For what's worth, on high, 27b is better, but both models on high are too slow to be used (way too many output tokens).

On low, 3.8 Flash Next is better AND considerably faster.

My results[0][1], on a RTX 3090 + 128GB DDR4 RAM. I still have to see if I can optimize any more settings for either, but I think I will switch from 3.8 27b to 3.8 Flash Next for my local LLM uses.

[0]: https://aibenchy.com/?q=qwen+3.8+27b%2C+qwen+3.8+next

[1]: https://aibenchy.com/compare/qwen-qwen3-8-27b-low/qwen-qwen3...

In my tests they do quite similarly, but 3.8 flash next is considerably (2x) more efficient and faster to respond.

Both 27b and flash next are more stable on "low" reasoning, only for generative /creative tasks, xhigh could be better, but both suffer from way too much reasoning at xhigh. And neither really support high, so low is the best reasoning effort.

[0]: https://aibenchy.com/compare/qwen-qwen3-8-27b-low/qwen-qwen3...

Thanks, trying it now. On my 3090 it seems to run at around 60tps, the IQ3_S variant. I am testing it now to see if it is better than Qwen 3.8 27b

> I still have to deal with it the same as I always have.

For me it was very different. Once I considered possible audhd, and knew what my triggers are and what is expected or not, my life is a lot easier and happier. I don't have to pretend anymore, and I no longer feel bad for being myself.

I am ok sitting in a corner in a crowded social environment and not feel weird about it. Even more, I meet more people who feel the same, so it leads to better relationships and more easy-going conversations, instead of forcing myself to be social for example.

Imagine some people have been calling you "crazy" all your life, in a bad way, and then, at 40 years old, you do something that you expect the other person to think is crazy and they just say instead: "ah, don't worry, that's normal to do if you have adhd", and then it suddenly clicks that you are NOT crazy and have never been your entire life, you just had a condition you didn't know about and people were ignorant, and acting like this is completely normal given the condition.

It's like being forced to write with your left hand the entire life, and suddenly you realize that you are right handed, and it's ok to be right handed too...

You can make a basic one in minutes based on existing open-source models.

Latency won't be that good, but could still work similarly. Simply force the structured output of a LLM to the given schema.

Probably also easy to train because we can use stronget LLMs to generate input/output data, or even synthetic data is easy to generate.

It's not really a new technology, it's more like a new use-case.

So, not using stuff like ChatGPT Dots?

Or using it in a way that's replaceable I guess, not having your entire work only ther, as your main machine.

I am also surprised by some results, but I also don't favour any model, so all models are tested exactly the same, and some simply fail some tests: incorect answer, requests timing out, not respecting output format requirements, writing code that doesn't generate the correct response, etc.

The top models, when used in a harness, will likely catch those errors or somehow manage to get to the right answer at some point, but those tests test mostly how likely a model is, when used via API, to give the correct response.

The benchmark is a bit sarurated at the top, there it's more relevant to see cost/response times/efficiency.

Opus 5.5 is not trash, still in the top models, but Anthropic models always have struggled with instructions following and refusing to answer questions. That being said, most tests are basically questions or simple tasks, and models could either do them ok or not. Nowadays the models and how good they are in practice is given more by the harness, than the model itself. I do think I should probably find a way to test the models including their harness, and to do so for a complex, long-running task, the so called "agentic" use-case.

Also, Gemini models have the best all-around knowledge, they are way above other models in general knowledge and domain-specific knowledge. No tests have web search enabled, and most modern models are indeed optimized for that use-case nowadays.

tl;dr: the leaderboard simply shows, given any simple question or programming task, which model is most likely to get the answer right.

I tested this model, it's ok, but too pricy.

It's luna level, but 14x more expensive:

https://aibenchy.com/compare/anthropic-claude-sonnet-5-5-hig...

We came a long way from 4o-mini: https://aibenchy.com/showcase/?q=Gpt&page=3#showcase=ac3f5db...

Really good Hamster[0], and in my tests it's very close to Astra, and indeed ~5x cheaper.

Actually, it is so close to Astra, that I'm starting to beleive it's a slightly quantized version of Astra.

[0]: https://aibenchy.com/compare/openai-gpt-6-1-sol-xhigh/openai...

I mean, Gpt-6 Luna seems a lot better in every aspect:

https://aibenchy.com/compare/fireworks-ember-1-high/openai-g...

I don't know, I have a weird feeling that Ember-1 is a distil/combination of Qwen + Gemini 3 Flash.

Looking at the hamsters drawings in this comparison, never saw any other model make such a similar version:

https://aibenchy.com/compare/fireworks-ember-1-low/google-ge...

Hmm, my phrasing was ambigous, I didn't mean it's one idea fo one fully functional app.

But more of, one feature idea, that the AI can add to an app.

So it's not prompting "a gamefied piano learning app", but adding a single new feature like "add a minigame, where the user can control a character on the musical staff, [... 20 paraphs later ... ], make a good implementation plan, implement and test the app until everything is production-ready and bug free /goal"

That is a single "idea" that also requires a lot of fine-tuning after, but rarely in terms of code, most of the times it's only in terms of functionality and design choices.

Firefox had an unsupported missing WebAudio. It should be fixed now, and I've updated my workflow to also test on Firefox. Let me know if the crash was in a different browser (not Firefox).

> I’m only saying that you can’t ignore the code for anything beyond a toy app

I thought the same, but I do disagree with this now. Future software development will be mostly just creating black boxes and describing what the box should do, without ever caring what is inside the box. For now, we still have to guide the AI and tell what architecture it should likely use, or which libraries should use (just for the sake of performance and ease of development), but in the future this probably won't matter either. I honestly believe code does not matter anymore. What matters is knowing what to test when building an app, how to define performance metrics and knowing what is good/possible when developing an app. I know I won't be able to create a shooting game that supports 1 million CCU on a single vCPU. But I know that the input latency should feel good, and game should run smoothly at start and over time, to tell it to implement tests to check for memory leaks and avoiding JavaScript GC pauses, use object pooling when possible, etc. The complexity moves from telling how/what to code, to knowing exactly to tell it how things should behave and what's a good outcome. If I tell it "make sure bullets are object pooled", it will likely implement it properly, as object pooling is a very common pattern that exists in its training data, and it usually either works or doesn't, when it doesn't work there are obvious issues like objects shown at the wrong positions or not spawning properly, so the issues with the code would be reflected in e2e testing anyway.

2. That was both motivating and demotivating to be honest. The app was my "baby", having spent a lot of time designing everything, optimizing, carefully choosing libraries and make cool implementation decisions. Now I feel like all the newly added features are not really mine, and it doesn't even feel like my product anymore that I can proudly say: "hey, I wrote all the code for this app". It is a really demoralizing feeling, but at the same time, I like how all my ideas can now be materialized. And it actually works. And it works well. I am still getting used to the fact that I won't have full control or understanding of how the code works, and it pains me that this is the case, but there is no way I could achieve better results by manually coding. I would rather have a feature having 90% of the ideal possible performance and UX, than not having that feature at all. Plus, that 10% is still doable, it just requires a bit of testing and asking the AI. The problem is that most of the times that effort is not really worth it, not for me, not for the clients. There are a lot of other low-hanging fruits that must be addressed, and that's how software development always worked. Now I am happy that I can actually do nice things that before I would have never spent the time on, like making sure a specific settings menu has better UX on mobile (before I would have probably just made an element smaller to fit mobile, for ok usability, but now I can tell it to design an entire new UI tailored to mobile for that specific feature).

It is definitely addictive, as it comes with instant gratification, as opposed to slowly coding and spending hours before you see any results.

I agree with your linked post, and that is sort-of a big issue (even though, most of the times, those guards are not there for the functionality itself, but for the AI, so that in case things break, it gets a more clear error of what went wrong and it knows how to fix it better). This is my entire system prompt, which I assume fixes some of that defensiveness issue:

    # Engineering style
    
    - Prefer simple, readable data flows and strong invariants over layers of defensive checks, fallback branches, assertions, and recovery mechanisms.
    - Validate at real trust boundaries, then let well-typed internal code rely on those validated contracts. Fix the source of invalid state instead of spreading null checks and guards through consumers.
    - Keep code concise and add useful comments that explain intent, ownership, or non-obvious constraints. Do not add speculative protection, tests, or abstractions without a demonstrated failure mode or requirement.
    - Use ASD-STE100 Simplified Technical English when you talk to me or caveman-like concise speech, be really direct

Firefox?

Thanks, only tested in webkit browsers, didn't know people use Firefox on mobile either.

EDIT: I can reproduce the crashes, thanks, will tell the AI to test on Firefox too, testing on Firefox mobile specifically might be trickier to debug actually.

Thanks!

> I’d expect you to have hundreds of apps of that level of complexity in a year.

I am working at around 7-8 projects at the same time. The limit becomes me having to remember what I was doing for each one. AI can implement things nicely, but it really sucks at deciding which features and having its own ideas about novel game mechanics or UI/UX patterns.

I don't consider it a "software factory", just a more robust way to implement features, and it's still a WIP. One system I implemented locally is called "TaskHub", which receives some implementation or testing details from a SoTA mosel like Astra and implements it locally using Qwen 3.8 27b running on a RTX 3090.

I delegate things like app testing, navigating the app and taking screenshots, analyzing screenshots, etc.

And yes, this specific app doesn't have any users yet, I will launch it next week after I finalize the user authentication, payment flows and mobile apps. I will release it mostly as is, see user response, and then change it accordingly.

I do have paying users for https://uxwizz.com

I only started using AI for development for this product (13 years developed without AI) a few months ago, and customers are really happy with the changes.

I managed to implement feature requests that were pending for years. It took probably 1 month to implement what would have taken 1 year without AI.