Is the safe fragile enough that anyone can break it with just their bare hands? If not, then what if you need to call 911 while your phone's in there?
You know though, you are mostly using the safe in your own home or office, for increments of 30 minutes to a couple hours. For many folks I think the risk is very low.
I guess, I find this emergency access sentiment really similar to something my wife's students say when there phones are taken away: "what if my parents need to contact me in an emergency?"
It's a really sincere worry they have that seems totally foreign to me, having grown up in an era when my parents often couldn't contact me for hours at a time. Food for thought.