* Wanting niche features that don't benefit other people than those in the enthusiast core, thus preventing the company from gaining market share and revenue.
* Ever-increasing expectations in terms of visible feature delivery (e.g. e10s was widely seen as a failure despite being foundational to move off a single thread model and increase browser responsiveness).
* General conservativeness in terms of anything that breaks workflows (famously [1], but also see the criticism of Firefox redesigns over the years, etc.)
* Most importantly, lack of proposals for monetization from said audience (donations do not cut it and smaller and more important projects such as OpenSSL, etc. have also been underfunded from time to time, so nvm funding a browser's development), while also opposing the typical monetization mechanisms, e.g. ads.
These things end up constraining a company from spending more resources to improve a growing product, as they don't have any. While more capital-intensive industries such as phone manufacturers often just choose to appeal to mass market at the cost of giving up their enthusiasts[2], Mozilla always wanted to hedge its bets, and has failed to go in either direction.
Therefore, it is not unexpected that Mozilla is failing, and only survives through whatever meager donations come through, and revshare from Google by placing them as the default search engine.