Speed isn't really a big deal for me. I want good quality code. It's already able to generate code 10-100X as fast as I could code it myself.
Anyway, are you speaking of the harness? The harness on mine isn't AI, so speed just isn't an issue.
Anyway, are you speaking of the harness? The harness on mine isn't AI, so speed just isn't an issue.
Chat Jimmy runs ~300X faster than the ~50 tok/s you are used to. What could you do differently when you are able to generate code 3,000 - 30,000X as fast as you could code it yourself? What if it was all good quality code? What would you do differently if it were 100,000X faster? mtok/s? gtok/s?
use the big models to code an adaptive small model. train it to use and build tools. give it a standard temple language for any project and bake it into a chip.
right now, LLMs are great because they dont need much data pruning, but once they break through to the functional components, the first thing to do is train a well scoped harness builder.