I know it's not the same as a commercial airliner, but I used to teach hang gliding on the beach and we'd sometimes get a little packed sand in the back of the keel (which is the name for the central bar that the wings were mirrored across). The total weight of the student plus the glider could get up to ~280 lb, and the amount of packed sand would be less than a pound or two, but the flight characteristics of the gliders would change drastically, to the point where we could barely get them in the air.
As another commenter notes below, the few people moving around the cabin is less about control ability and safety than it is about fuel efficiency.
It is not about balance.
Moving a few people probably doesn't cause an issue if the rest of the plane is completely empty.
If you move 20 people from the right side to the left side of the plane this will probably have some kind of effect.
On smaller regional aircraft they sometimes ask people to move seats to the other side or back to front.
They rarely upgrade to international first/business because the meals and other perks cost real money. And it'll piss off the people who payed over a thousand dollars for those seats.