Things that could go wrong: The end-user's frontend (a 4-year-old phone browser, a crappy smart TV, a Chromecast, Microsoft's browser abortions, whatever) has to support your protocol of choice. HLS is more and more common but it's not a guarantee, whereas the Vimeo and YouTube players have years of legacy and fallback support and have their CDNs pretty stabilized. With HLS you might still need a frontend JS player and then have to manage the distribution and packaging for that. And if you care about the video player UX, things like supporting user-chosen resolutions (overriding network speed), theater mode or easy full-screen or subtitles, 1.5x+ speed, etc. And you have to invalidate and re-CDN a bunch of separate files every time your source file updates.
But really, again, it's not that you CAN'T recreate what Vimeo and YouTube etc. have done, but why would you want to? It's a lot of effort to end up with an inferior UX, higher bandwidth charges, and a whole new pipeline (if not stack) that you have to maintain. No reason to reinvent the wheel for such a common use case.
Edit: And you STILL have to CDN it in the end, unless you have the phattest pipes in all the worlds. So you still have to subcontract with a third party in the end. Why not just put it there to begin with? It's silly.
It's one thing if you're Google-scale and need to in-house everything because you're dealing with a billion viewers a day, but when it's just a few thousand (the OP said that in a different comment), it's really not worth your time or bandwidth charges to reinvent that infrastructure.