If your phone gets stolen, then you block your phone number with the carrier, and the thief can't get your SMS - so 2-factor security still works and protects you.
But in a worst case scenario how quick could someone determined actually compromise you? minutes? less time than you'd wait on hold during work hours, let alone waiting till the next day?