Yes! You do!
So scissors hinge on a pivot point, and they hinge in two dimensions. They mostly move in the dimension that they cut in, but they also move a little in the perpendicular dimension. And in order to bring the cutting edges closer, you move the handle ends farther away. If you're using the correct-handed scissors, doing so is easy and natural because your hand grip curls one way easily. If you then switch the same scissors to the other hand, it's hard and uncomfortable because your other hand is a mirror image and your natural hand grip curls to bring the handles closer and the cutting edges farther away.
The fact that you keep mentioning that this violates some law of physics is bonkers.