This is an excellent point, but I think it's always worth considering that the tradeoff isn't between third-party security vulnerabilities and no vulnerabilities; it's between third-party security vulnerabilities and first-party security vulnerabilities.
It's up to your team to decide if you have the capabilities in-house to deal with those. If not, trusting to the wisdom of crowds may often be a better solution. (At the very least, if you're rolling your own crypto solution and not publishing it, take a strong look at why you're doing that and be very sure you aren't falling victim to smartest-kids-in-the-room syndrome).