Just added a website for the same benchmark, easier read: https://aideploybench.com
402 karma · joined March 23, 2014
Previously RStudio, The Linux Foundation, Microsoft Research, and SAP.
x.com/javierluraschi github.com/javierluraschi linkedin.com/in/javierluraschi
The benchmark runs five web apps: (1) static website for say a company page or the like (2) the usual CRUD with auth for say SaaS product (3) an API to be consumed by other services (4) a dashboard with data to mimic internal company dashboards and the like (5) a realtime chat application with support for web sockets.
It currently compares four platforms but can be easily extended to more, feel free to open GitHub issues or send me a PR.
I was specially interested in comparing human intervention (Human Time). As in, how much time is spent for a human to deploy to particular platforms the first time?
The benchmark tracks build/deploy time, tokens, agent cost, security, compliance review of the deployment (secrets handling, least privilege, terms fit, attack surface, etc.). Methodology and reports (including screenshots) are in the repo.
Repo: https://github.com/cloudishai/ai-deploy-bench
Full disclosure: I'm working on Cloudish. Happy to answer questions about the methodology or results.
Edit: Removed table due to formatting issues.
https://www.linkedin.com/feed/update/urn:li:activity:7298354...
Puts in perspective how far we've come in just one year.
So if you can work 10% more than your peers, you get not 10% bonus but rather 30%-100% more. So it makes business sense to put the extra 10%, until everyone is working at 110% and then again, adding an extra 10% pays off, rinse release, death spiral.
The compensation model is pure evil.
Then plot twist, I turn the non-profit for profit, I avoid releasing the cancer cure I promised and instead I charge you for cure.
If a non-profit promised to cure cancer to help citizens, I would want that at the very-least they remain non-profit.
https://news.ycombinator.com/item?id=43450044
I’d very much argue that I vibe-coded a lot, but also spent a lot of time reviewing and optimizing permissions, testing and code reviewing it. Very much a practice I would personally encourage our engineering team to adopt.
“If an LLM wrote the code for you, and you then reviewed it, tested it thoroughly and made sure you could explain how it works to someone else that’s not vibe coding”
You can have a valid workflow were you vibe-code, and then review it thoroughly, that’s still doing vibe-coding + good software engineering practices, but I still consider it vibe coding.
That said, I recognize that maybe we are just debating definitions.
But I would like to argue that vibe-coding can be a good coding practice on its own when properly integrated with sound software engineering practices
Spending 200K man years for a tomb seems like a pretty poor use of resources.