EDIT (comparison)
Very impressive, and a lot more work than it looks from the outside (the DAgger loop especially). I built something similar in June, vibeSH, a shell that hallucinates the whole computer, but from the other direction: no training, just a frontier model and a long prompt, so maybe the comparison is useful.
The granularity is different though. You model keypresses and predict which rows change, I batch keystrokes and have the model repaint the whole screen each round trip. Non interactive cli programs work very good, but interactive cli programs like vim, less and top do work, but it feels like SSH over a 1996 modem, you type a burst, wait, and the screen catches up. Yours is going to be a lot more responsive at that.
The big model buys consistency, for as long as the session fits in the context window. Files come back the way you wrote them, python output is usually right instead of guessed, nested paths stay where they are. The price is an API call per round trip and a cost that grows with the session (and it's not 208 MB and doesnt run offline).
Mine offers an @ai prefix that talks to the director instead of the machine, so mid-session you can say "@ai make this a haunted Solaris machine from 1996" and the filesystem, the motd and the error messages all change to match, and the directives stick.
Your hardest problem and mine are the same one I think. For you it's a 360M model losing track of state, for me it's the context window filling up, after which the box gets amnesia and old files have quietly rewritten themselves. I decided early not to fight that completely and made it canon (the box has bad memory), your DAgger setup is an actual fix for it, which I don't have.