I Use Development-Driven Specs, Not Spec-Driven Development | Hacker News Reader

I Use Development-Driven Specs, Not Spec-Driven Development

github.com

20 pointsWaywardGeek26 comments

One thing I noticed when I first started with the harness and local LLM was that how I was writing the initial specification needed to change. By watching the thinking of the LLM I gradually honed the specification of what I wanted until the point where it was able to one shot it correctly. Then that spec fed to deepseek, GLM, smaller LLMs even like Ornith 1.5 they all managed to one shot it but with clear differences in the quality of what they did.

I definitely found value in this approach by seeing what the LLM did right and wrong was a large part about what the specification said. Fixing those issues also reduced the LLM thinking tokens. The specification was better. This is a similar style of process to what is used here, but really learning to adjust to what the LLMs do well and don't do well from interpreting the input seemed to be key.

One thing I wonder is if hanging onto all these altered specs might actually be a complete system definition and if I just keep piling them up one day I could feed this to a substantially smarter AI and it will just build a better application from the same specs. Its potentially more work than its worth to always do this process of updating the spec especially if its only a few issues to workaround.

My everyday work is with a very legacy framework, and I've noticed that for the best results, I end up smacking on so many skills, additional tools, and so on to the agent, that at this point it's a harness specifically designed for this project.

While this was the result of necessity, I think it helps a ton to have the harness fit your specific dev experience, like an IDE full of the specific plugins you'd want - just not an ide, because ides are OVER.

@dang, moderators - IMHO the linked repo is sufficiently interesting to have the flag removed

This results in an executable specification: give it to an AI coding agent like Ensemble, and it will execute the specification autonomously, creating a new version of the code. This works perfectly when I create new versions of Ensemble by executing "The Art of Building AI Coding Agents". I've never seen a more successful specification.

You have to build the spec as you build the code. The processes cannot be separated. I do the usual up-front design conversation with the AI, and then ask it to write just the TL;DR and exercises sections of the next chapter. After that, I pair-program with the AI to implement it, following the TL;DR, writing notes as we go to clarify what was unclear, steering the AI in real time towards my design.

Once the new code passes all the exercise graders, only then do I ask the AI to read the notes, voice.md file, and write the body of the chapter. Then, I have it launch a student to see if it can write the code purely from the chapter and graders.

At the end of this process, I'm the expert, and I don't even need to review the code.

Code is simply the result of executing the right specification. Just another artifact of the compiler.

Stop vibe-coding. Become the expert by pair-programming with the AI as you build to code, and then build the specs from what you did to build the code. Don’t document the code to build. Document the process, lessons learned, and why you built it this way. You can’t do that up-front.

If you don’t know how to pair-program with the AI, well, you can always try running Ensemble, and listening to the AI as it works. It won’t call tools until you’ve listened to every word the AI says, and the AI isn’t allowed to silently call tools.

I know this is hard for all of you folks with good vision. Get over it. Just do it.

It would be very helpful to show a demo of how you're using Ensemble to build an app, say a business app and what is the final result. I'm not sure this is what I imagine it is but it appears interesting concept. I'll remind myself to try it next week but if I saw a demo that would be a more definite attempt at it. Are you planning to publish any demos?

I'm happy to publish demos. What sort would you like? It is optimized for me, an nearly leagally blind SWE who listens at 5X speed. I process information almost that fast.

However, Ensemble can be customized to your needs as easily as I customized it to mine.

IMO, the most impressive demo takes about 3 sayds, where I give the book and graders to an AI coding agent, and it builds a new AI coding agent from the book. It does take 3 days, but IMO, is a hell of a demo.

I'm not blind but can't say that watching a demo of your workflow isn't intriguing and am sure others who are also blind would find it useful. Do publish it if you can. I'd also be interested in a more traditional demo. Maybe not a full demo but I need to make it click for me so I can start off some of my own experimentation. Thanks

> You have to build the spec as you build the code. The processes cannot be separated.

> This results in an executable specification: give it to an AI coding agent like Ensemble, and it will execute the specification autonomously, creating a new version of the code.

Why do you need to create a new version of the code "you"'ve just written?

Two things

1) The quality has increased significantly each time. 2) Until Ensemble regenerated Ensemble, the loop ahd not closed.

I cannot tell if this is a parody

It's actually genius:

> Code is simply the result of executing the right specification (...) and then build the specs from what you did to build the code

You might make a profit selling paperclips...

"The book that builds an AI coding agent. The agent that builds the next edition."

Really?

Yes. Simply clone the repo and give it to Astra, Opus, or Fable. Really.

https://en.wikipedia.org/wiki/Generation_loss

It's real. Each iteration through the loop, is the quality increasing or degenerating?

That's partly why I switched from Go to C# in the 3rd edition (third time executing the book, including the first time when I pair-programed with the AI to created it).

The quality of the codebook is either improving ore degrading. I believe so far, I've managed improving.

a collective madness or mass IQ collapse

It isn't IQ collapse. Madness, possibly.

based. Please, someone unflag this man's thread.

Is this such a complicated concept that you had to have AI generate a book about it?

Since you mention it in your repo, perhaps the more interesting discussion is about how you work with reduced vision. I noticed you use a screen reader: https://www.youtube.com/watch?v=IGtAZGLtvXw

Sorry, you did ask about working with reduced vision. IMO, vision is a distraction. If you truly want to pair program with the LLM, you have to read or listen to 100% of its thinking. When I pair program with a junior SWE, I always ask them to say what they are thinking. It is no different with the LLM, except it is really helpful if you can listen at hight speed. I listen at 5X speed.

I built CodeRhapsody (an internal Google project) for me, to make it possible for me to create at the speed my brain works.

Here's the honest story, and I don't fucking care if I get fired for it.

I tried to open source CodeRhapsody, my awesome AI coding agent that I built with the AI to help me and other low-vision coders. Fucking TK, or VP in charge of Cloud hates giving anything away for free, so my weak and stupid directors and VPs rejected it.

So, I wrote Ensemble. It is not only better, but it is forever. They can claim ownership of the code, fine. It only takes 3 days to execute the book, and create an even better version of Ensemble.

So, yeah, it talks to me, exactly how I want it to.

Ah, I see my post is now "flagged". I assume from profanity?

Yeah, I'm really not sure why you got flagged either. Reach out to @dang to unflag.

Elon just paid more than $60B for this knowledge, which frankly is why I decided to publish it for free. Yeah, it's a bit complex, but we're smart, right? We can learn anything.

I spent a significant part of the last 18 months learning this at high speed. You're welcome.

source code is the spec, kiddos

Just the opposite. Source code is an artifact of executing the specification.

Our roles change. We no longer write code, but instead encode knowledge, which the LLM is incapable of learning.

Yes but considering a counterexample. You inherit a production codebase with scant documentation form someone else who is no longer around. In that case the source code ends up becoming the spec more or less. How would you use ensemble to extract the spec form an existing project?

Reply on news.ycombinator.com