The bank I use maintains a list of people I allow to access my box along with their physical signature. When I needed to access my box, I had to sign in with a pen, on paper and show my ID. They compared that signature with the one I gave when I first obtained the box. I was granted access if they matched. If someone else came in with the key but their name wasn't on the bank's list or the signature didn't match, they wouldn't allow access to the box.
So make sure people you want to be able to access the box are on that list (which means they will have to go to the bank to provide a signature ahead of time.)