Or pretend to have lost their passport and get a new passport without stamp. Or get a secondary passport that's made available by most governments because of issues like this one.

But you are right in principle. Letting people through because they look important is a big security hole.

There's not a whole lot that border control can do to stop people who are good at making fake passports though.