1) The unpleasant interactions, while rare, tend to be VERY unpleasant.
2) Unless you are solely devoted to working on that thing as your only hobby, people start wondering about the delay between code/releases.
3) A certain amount of the work is just keeping the lights on: making sure your dependencies are secure, making sure that new change in Github actions doesn't break your build, etc. If you only have a limited time to work on a project, a lot of it gets spent here sometimes.
4) The more popular something is, the more eyes on it, and the more bugs are found (and the bigger chance your bug fix introduces new bugs, especially if you have limited time to spend fixing it). This then takes up a larger percentage of your time.
For a certain level of open source popularity I feel like I only get a few options. I can turn it into my job somehow, I can make it my only hobby, or I can walk away. In a few instances I've chosen to walk away. Sometimes there is someone to take over, sometimes there isn't. No matter what it feels like an impossible choice.