XCSme

No, the problem is not the code.

Is that that variable might be used in tests, UI, docs, agenr markdown files, other related projects too.

The design spec could say "always use 10px margin", then in code we have a const with value 10 used everywhere. If we update 10 manually in code to 20, then the design spec is now outdated.

> The critical part is providing it a way to validate its work end to end.

Yes! Part of implementing it "one-shot" is actually a loop of planning, implementing, testing e2e on various devices and various edge-cases. It's not just writing the code, and this is not how "vibe-coding" works nowadays. It's not just writing code anymore, this is why most providers worked on their computer-use support too, not only for doing tasks, but also for being able to check and test the work they do e2e.

I just asked the model to implement 2FA and it did address all the points you've mentioned.

EDIT: Support was not the case, because the app is self-hosted, but the docs should indeed provide instructions for recovery if both authenticator and recovery codes are lost.

- handle accounts that have lost their second factor in an way appropriate for your business

One-time displayed recovery codes are a standard practice, and most good LLMs will add it by default without even asking for it.

And yeah, I am talking about TOTP apps, I think sms/email is not as secure.

How can we write without thinking?

If the problem and implementation is so well defined, and "determnistic", this means that LLMs should also be able to just "write code" from specs, without thinking.

2FA is quite simple, right?

It's just a secret key that an autheticator uses to generate a time-based code, which the app can validate before completing a normal log-in flow.

What model did you use?

Astra xhigh on fast mode can probably indeed one-shot that in 5 minutes.

Plus optionally QR image generator to easily add that key to the authenticator app.

There are already many libraries doing 2FA, but implementing it in any language is quite trivial, right

It's a heurisric though.

It's one thing to test once and find 10 bugs, and another to test 10 times and find 1 bug.

What helps is to ask the model to make design docs and to note down features and how they work in md files. It sort of provides another layer of persistence separate from code.

But yeah, overall you have to be ok with the app being approximate too. Maybe after an update a button is a different size, or in a different place, or it suddenly has an animation to it. Those smaller things are a bit harder to control when making changes at scale, and if not clearly documented.

For me this is not necessarily a big drawback, for things like games, the core game loop, performance and game feel are a lot more important than any small UI tweaks.

Hopefully, the better the models get, those side-effects will only be improvements, not degradations.

Yeah, I agree, they are far from perfect and make a lot of mistakes.

Truth is, modern software was already quite shit and full of bugs. All major apps had bugs, issues, going down, etc, so users did get used to things not working. I honestly beleive AI coding nowadays, for better or worse, does things better than the average developer.

Yes, it is overly defensive and verbose, but the end result is in general ok and fully functional. Yes, it adds 30 tests and "release gates", and they are not even that useful, most of the times they just act as an extra safety mechanism to make parts of the code immutable, so release fails if the model accidentally changed things.

Another issue with looking at code, is that it's very hard to manually change things anyway. I can't just change a variable from 10 to 20, because I don't know where it is used. I have to ask the model to set that value to 20. It is quite stupid and inefficient, but this is one cost of coding using AI. But, if you do this, things will likely work.

That being said, I've mostly used Astra xhigh since it was released and things just work.

Thanks for your words.

> If the core or that app is more than a few thousand lines of code, something is seriously wrong.

That's the thing about vibe-coding: it is not the core, it is the entire app. We no longer make MVPs and release those, with AI we can make directly the app including all bells and whistles, entire progression, not just one level, all the systems around it.

Why? Because if something needs changing, it's just one prompt away. I do think code is fluid now, any choice of architecture can be instantly changed at basically no cost.

Maybe my mind is just finding ways to cope, thinking that I "wasted" thousands of solving coding challenges and fixing bugs, but I do think, for better or worse, that manually coding is gone. Same as we no longer code in assembly anymore. We no longer write C. We no longer write JavaScript. We no longer write TypeScript. Maybe not today, people don't like change, but manually writing or even viewing code will only be done in a few educational and high-performance/risk cases.

I make my own software, so no team to look over the code or be bothered by it. I don't even know how "vibe-coding" works in a team environment, because for me now it feels like it's "ideas to app" directly, so dumping by brain/ideas directly into a functional product.

I think this one is really cool[0], will be a free piano learning app. I do have other projects, but they are all at around 80% too, because some systems are shared amongst the projects and have to be finalized too (i.e. now I'm implementing my own transactional/marketing email service on top of Amazon SES, I need it before releasing ultimidi so people can register and receive email confirmations).

[0]: https://game.ultimidi.com

No, it was about testing only. I will only do a full test of the app when it's ready to be released, otherwise it makes no sense, as the app/product/flow still changes a lot. It's like prototyping a game, you try it a few times, see what's fun, what's not, and keep iterating. Only once you think it's good enough, you can start fully testing and polishing it.

Even after release, I don't see reasons to check the code if everything works and people are happy with the app.

Interesting, what browser are you using?

Also, what do you mean punch in notes? Like mashing keys and pressing 10 buttons at once?

Never had the app/website crash, tested only on Brave (desktop and mobile) so far.

Sort of, it's more of a heurisric of bugs found so far. There are very rarely any bugs, most of the changes I request are for tweaks/logic changes, not bug fixes. Usually things just work.

Also, I barely tested and kept changing things simply because of this: whatever I ask for, seems to work as expected.

I could probably look over, and get some snippets, but currently only the client-side is live, so the code is sort of there, albeit I assume it's minified for production.

The game seems quite bug free though, including minigames. The UI could be better, but it's not done yet.

I do for example have an automated system that simulates progression, takes screenshots of the game to find potential hidden buttons or overlapping elements, to test for performance, etc.

If it looks like a duck, and quacks like a duck, I honestly don't see why I would review 100k's of lines of code.

EDIT: I might have replied in a wrong thread, but it was about this entirely "vibe-coded" app: https://game.ultimidi.com

Hmm, my experience is different. I noticed Astra always asks me to add credentials in a secure way, to never pasted them in chat, to rotate any potentially exposed tokens, etc.

Can you give an example of unsafe defaults used?

This app[0] is entirely "vibe-coded", started on bolt.new which was like the OG way to vibe code and slowly improved over 1-2 years using various models.

I have no idea how the code looks like, and barely even tested the app entirely, because it is still not released yet, but I do think a lot about new features, tweaks, improvements, etc. Years of programming and game development did help, but I don't think anymore that code is relevant, as long as it looks ok and feels good.

[0]: https://ultimidi.com

I think it's the opposite, with the right guideance, testing, frameworks, and using the top models today, implementing something with AI is most of the times better than what most developers would do.

I basically only manually test e2e myself, other tests are automated, code review is automated. I can tell the model to test for me too specific things or to add tests for specific potential issues, performance benchmarks, compared different implementations, etc.

The focus is a lot more around the code than on the code.

No.

Haven't typed a line of code or read any code for over 6 months now.

And I used to love coding and be a competitive programmer, but this is how "coding" goes nowdays.

I have a mental model of what it would do, and how it would work, and I ask questions to confirm things and tell it to watch for specific gotchas. Then simply test the feature myself a bit.

Security-wise, I think the latest cyber models are better than me anyway at finding vulnerabilitates and pentesting such features.

Plus 2FA is a very common pattern, so it likely has in the training dataset many really good implementations.

Wriring the code is not the challenge, but it's what was taking up most of the time. Not the typing itself, but also because I had to think of how to implement it.

Now I can just say "add 2FA" and in 5 minutes, while I test something else, it is done.

It also made iterations a lot faster, you can try something out, see how it feels, if it doesn't work, you can just trash all the code and start again.

Also, Terra is gone, GPT-6 Luna is smarter than 5.6 Terra and costs *15x* less [0].

[0]: https://aibenchy.com/compare/openai-gpt-5-6-terra-high/opena...

It's an estimation based on information found on the internet/rumors: https://aibenchy.com/model/anthropic-claude-opus-5-5-high/

Mostly based on previous relesses info.

A good improvement overall.

GPT-6 Luna now is 50% cheaper, which makes it have one of the best intelligence per cost ratios.

GPT-6 Sol is smarter, but seems to reason 2x more than GPT-5.6, which makes it 2x slow3r and 25% more expensive in practice.

[0]: https://aibenchy.com/compare/openai-gpt-6-sol-high/openai-gp...

GPT 6 Sol is still better and 2x cheaper in my benchmarks:

https://aibenchy.com/compare/anthropic-claude-opus-5-5-high/...

I use a lot of ChatGPT remotez and 70% of the times is unusable and buggy (prompts disappear, a lot of errors, buttons don't work, etc.)

Mimo scores higher, even though it got more questions wrong, because the coding tests are scored higher than other tests, and Grok failed some coding tests, which is surprising tbh, as they aren't that hard.

The timeouts are my self-imposed limits for the test.

16 minutes per test, which is a lot for simple questions...

Sometimes they fail because they reason more than their max context window without giving an answer, that's odd too.

It seems to be better than Grok 4.7 and a lot cheaper in my tests:

https://aibenchy.com/compare/x-ai-grok-4-7-medium/xiaomi-mim...

I tried them, but could only test Pro none and Flash none and low, the other ones (medium/high) used way too many tokens and all requests timed out. Not sure if they have a problem with their API, or this model is really token inefficient/basically unusable.

That's medium, xhigh is not much better:

https://aibenchy.com/model/x-ai-grok-4-7-xhigh/#showcase=2f9...