Yes, I totally hear you -- this is why we treat any API unavailability as such a huge deal and feel so bad about it. Whether customers can or should retry is a question we sometimes get from users after API errors and so I wanted to be proactive about answering it. (And our users often have Stripe.js tokens or customer tokens that they can retry.) But, you're totally right; this definitely doesn't solve all cases.
In the unlikely event there's anything I can do to help, definitely drop me an email.