The vast majority of our users prefer PayPal (80%+). For one particular product we had to roll out using Stripe only, the number one customer service request we had was to provide PayPal as an option. Subscription numbers rose by over 50% overnight when including it as a choice (literally tens of thousands of people signed up after the change).
Internationally, PayPal seems like the de facto choice for making online purchases. In addition, outside of the Bay Area tech bubble, I believe people in the states are simply more comfortable with using PayPal over Stripe.
While PayPal gets a lot of hate, I'd think long and hard before deciding not to offer it as a payment processor. It's possible that you're leaving a significant amount of money on the table.