I started to experiment with the same idea on a small weekend project. I find it is quite hard to come up with a prompt that work well consistently. I built the thing inside a docker container for "safety" (there is probably a lot of improvement to make on that aspect).

Here is the repo if you want to take a look: https://github.com/antca/geppetto/ It's just a WIP experiment, don't take it too seriously, please. :D