First of an most importantly, it doesn't apply to like writing "please don't crumple me up and throw me in the bin" on a piece of paper, then crumpling it up and throwing it in the bin, which is what this LLM stuff amounts to.

But it would apply to making a human clone, which is exactly what this LLM stuff isn't. I mean, if it was possible; an "atom-for-atom" clone of someone would require you cloning a full universe around them. If they don't have the same life as you, they're not you, and they can't have the same life if everything else doesn't, too. We're not just a bunch of atoms on a discrete grid.

But if it was possible to just copy and paste humans like that, how would you know you're not a copy, and in 5 seconds you'll hear a voice "prepare for termination in 10 seconds"? You might argue it wouldn't be torture because you'd just check your browser tabs for 10 seconds to find the ad the voice came from, and then poof out of existence. But if you lived in a world where that was commonplace, you'd you'd be terrified and sad every bit as you yourself would be right now if you knew someone could just end you, and pressed the button to make it so.

Let's say someone told you the universe and time are infinite in some way we don't fully understand yet, but you knew 100% it was true, and so there wouldn't just be infinite variations of you, but also infinite exact copies. Would this make any difference how you would feel about getting mistreated or having to die? I assume not. It's not (just) about your qualities and being unique [0], but how that feels from the inside, so to speak.

[0] and everything is, even a perfect clone can't occupy the same space at the same time, so there actually can be no perfect clones, I would argue, and there aren't even two atoms that are exact clones.