Has anyone hooked this up to a unit test system, like

   LLMtries = []
   while(!testPassed) { 
      - get new LLM try (w/ LLMtries history, and test results)
      - run/eval the try
      - run the test      
   }

and kind of see how long it takes to generate the code that works? If it ever ends, the last LLMtries is the one that worked.

I haven't done this because I see this burning through lots of credits. However, if this thing costs $5k/year but is better than hiring a $50k a year engineer (or consultant)... I'd use it.