Except you're not. You're using a copy of their property without permission.

If I trespass into someone's house, it's invading their personal space. If they're in the house, it's invasive to them and alarming and would probably make them afraid that I intend violence. While in the house, I may damage things or cause disruption.

However, if I use a device to create an alternate universe with a perfect copy of their house, without them inside (since it's my own private universe), and then I enter the house and do whatever in it, I'm not disrupting them at all. I'm not inside their house, I'm in my own copy of it in my own universe, and they don't even know. It doesn't affect them at all.