Nice work! You may be interested in a paper that explored a similar concept and included some interesting ways of dealing with latency called WorldScribe: https://worldscribe.org.