At the risk of repeating classic HN Dropbox hubris... this just looks like what Automatic1111 + ControlNet is already doing.
How do these work together in hubris world? I'm always up for a bit of tinkering. I'll figure it out later either way but a head start never hurts