I really like that I can add boarding passes to Apple Wallet and they're available totally offline. There's not a lot Wallet is good for (adoption is too low) but it's great at Boarding Passes.
In any case, as the other person mentioned, my phone is locked the whole time. The only thing in the wallet they can access is my boarding pass.
Usually you don't hand your boarding pass to anybody. Instead you self-scan it once when you enter the passenger / transit area and a second time, when boarding the plane through a self-service gate.
This may vary by airport, but on most of my flights through a number of European airports access is pretty much automated.