XCSme

Yeah, it's hard to find a single simple task that all models fail on, in low context length conditions.

Also because models now are actually not that good on knowing things (domain knowledge), as they rely more on web search on tool use. So if I added a question, about some obscure fact, probably the SOTA models would fail it, but in practice they would find it with web search enabled. Not sure how to handle that. This is also why Gemini is on top, it's good enough at coding and instructions following, while having by far best general and domain specific knowledge.

I waited until NovitaAI provider became available on OpenRouter.

I have their guardrails enabled to not allow requests to providers that train on data.

As far as they say though...

What seems inconsistent?

The coverage is quite small, only 22 tests.

It's more to compare the cost/speed/consistency between models, given the same tasks.

https://aibenchy.com/compare/deepseek-deepseek-v4-1-flash-hi...

Seems just slightly better than last v4 release, considerably (3x) more expensive, but also faster and slightly more token efficient.

XCSmeon Muse – Meta’s personal AI agent

In a dystopian outcome, the goal of having more people use your AI will not be for them to pay for it...

XCSmeon Muse – Meta’s personal AI agent

I think all AI companies want to capture normie users too, it's a huge market.

All ChatGPT, Claude, Google, Microsoft have/want to have their own assistants.

Sounds more fun tbh

Can't you ask Astra to do that too? That's the promise, that it can use the computer as you would.

I do, and some clients did start using Claude, where they just say their problems and Claude provides the solution.

Yeah, ranking is for entire model, not only SVG generation. Maybe I remove it entirely and add generation date instead.

I was thinking to manually grade/rank the SVGs, but I decided against it, as it is indeed subjective.

I was thinking it could have at least a simple objective check (hamster doesn't have extra or missing parts, table has 2 sides, and net is in the middle, etc.).

This will only work for online/hosted software, right?

If local one can be one-shotted?

Or not everyone would care or know how to use AI...

That's true, if it really "knows" stuff, a bit weird to have basic flaws like 2 mouths, and sitting on the side...

https://aibenchy.com/showcase/?page=2#showcase=67fc6d6c8e4c3...

Well, this was the state of SVG generation until a few months ago:

https://aibenchy.com/showcase/?page=3#showcase=c215b5c915da6...

I've replaced "Score" there with model ranking, to reduce confusion, thanks for the feedback!

https://aibenchy.com/compare/openai-gpt-6-astra-high/google-...

Do you prefer the fable one?

It's more "correct" but looks a lot worse in my opinion:

Yeah, that's confusing, the score is for the entire benchmark, not for SVG generation only.

Good point about the mouths, I just noticed, lol

Imo, it's still better than most models, I personally like the stylized perspective.

You can view here all generations for all models: https://aibenchy.com/showcase/

https://aibenchy.com/compare/openai-gpt-6-astra-high/google-...

That's some crazy SVG generation:

It took a while to test it, initially OpenRouter was giving Not Found errors for this model ID.

I don't think AI can build anything blind that is production ready. Whatever results it gives initially are just a starting point for future iterations.

Almost all my projects are continously updated almost daily, based on my feedback and some automated systems (i.e. SEO feedback loops)

From what pros are saying after matches, especially in classical chess, it's almost always preparation and metagaming the opponent vs finding good moves.

It's a lot of mind games involved, going for specific openings, repeating same moves as last match, doing a poor by surprising, move etc.

Many GMs said that to win a match you have to play non-ideal moves many times, otherwise it will likely end in a draw.

XCSmeon The Raspberry Pi Interactive Timeline · 2006–2026

I basically stop caring about the code, never look at it, and just tell it what to be aware of from my experience.

Tell it stuff like "use caching", "use library X", "make sure that it works when offline", etc.

And then basically live test it and see if it works as expected, and tell it more improvements/changes to make.

Cool, I started using a RBPi4 at I had purchased 3 years ago and never used. I use it to be able to start my "AI-agent" PC remotely, to send wake-up-on-LAN signal from anywhere. I am also thinking now to add a tasks queue on PI itself, so I can send requests directly to the PI, it stores them, and maybe once or twice a day, it turns on the PC, sends all the tasks, then turns it off.

It is, if you are trying to sell all of them in the same market, so 99 stalls with mass produced stuff, one with handcrafted stuff, but you don't know which is which.

> You should be happy with your own creation.

I mentioned this in another comment, but we are social animals, we get most joy by sharing what we have created. And the problem now is that it's getting very hard to share and discover genuine creations vs AI-generated ones.

Yeah, this is also the idea of materialism in a way, that everything that happens is a consequence of what already exists, nothing new is ever created, just a permutation of the current state.

Still a hard philosophical, to know whether we have intelligence/free will, or just really complex algorithms that combine existing knowledge.

I very strongly agree.

I think the next 2-4 months are the best time ever to build and sell software. But it has to sell now. It will likely be obsolete in 1 year, and no one will care anymore.

I think this is the last easy chance to capture an audience.

Thanks for sharing!

I am so sorry for your loss.

I agree, some things should be automated, something should never be. I think it will take us quite some time to realize which is which, and the consequences involved.

The problem with AI now is that it seems to be taking over all the things nice, and making it harder to appreciate human-made things. It's like being in a museum with 99 AI-generated paintings, and 1 human-made one. Whould would enjoy visiting that museum? You don't even know which one is the human-made one. This is how it feels now, what's the point of creating "hand-made" stuff when it gets lost in a sea of slop? Both parties are upset (creator for not getting attention, viewer for not knowing anymore what is real or not, or what was made and curated by a human or not).

I made a random tweet yesterday, that "the future is offline". I do think we will get fatigue from having access to infinite, hyper-personalized content, so much that we will completely disregard ALL content.

I also lost my 31yo partner to cancer last year, and the only thing she cared most about was being remembered and knowing that she had done something meaningful in her life, that she left some impact on this world. AI is taking many of those opportunities, to make impact.