ControlNet is where it's at - the depth model can completely reskin an image [1] with a bit of prompting, and you can compose them (multiple controlnets feeding on each other)

[1] https://twitter.com/herval/status/1631113661207945217/photo/...