Since you guys are digging into the MediaSync work, you should look into Croquet (https://www.croquet.io). It's a novel synchronization framework for ensuring connected clients stay perfectly in sync and that it scales up appropriately.

I think it might make sense for the "multiplayer" aspects of what you're trying to build.

Check out the demos: https://www.croquet.io/demos . They have demos of synchronizing video playback, text editing, object manipulation, etc.