Ask HN: Any payment processor woes?

I'm about to choose a payment processor for my US based SaaS company. I will host the credit cards off-premise at the processor to avoid intense PCI audits, and take advantage of built-in recurring billing/subscriptions.

My two top contenders are Stripe and BrainTree.

I've worked with Vantiv, and Authorize.Net in the past.

What are some of the sufferings you've experienced with a particular payment processor? Any advice to heed, whether on API difficulty, or customer service?

All horror stories welcome.