We do, there are people making web browsers that do unique things or work in unique ways. Note that innovation in my view is trying something new, I don't believe it needs to succeed necessarily.
https://bonsaibrowser.com
https://polypane.app
To some extent Vivaldi is one as well. https://vivaldi.com
The brave browser integrates a cryptocurrency as a first-class citizen.
There are also other types of innovation on the JavaScript, html and http end.
At this point almost a decade ago, Chrome was pushing to add support to run dart natively in web browsers. There is still a chromium fork available that does this: https://github.com/dart-lang/sdk
As for internet protocols, people are definitely working on this as well. In the cryptocurrency world, we have ipfs:// and ens:// (ethereum name servers). I believe both of these have been implemented in the brave browser, but I'm not entirely certain about that.
Yesterday, there was an "alternative" to the HTTP protocol on the front-page of HN as well, gemini: https://news.ycombinator.com/item?id=30072085
----
I also really want to mention the recent work by a HN developer on marginalia.nu. It's not directly web-browser related, but creating a niche search engine is related enough!
----
The reason for why no new engine is coming clear, rendering css and html correctly is a gargantuan task. The last successful attempt was Chromium. Mozilla tried this with servo, but it ended up failing due to what I believe was that Mozilla believed it was too big of a project and costed too much.
It's not impossible, it's just something that requires large resources and a lot of time, not dissimilar to developing a real-world operating system from scratch today.
----
I believe there are plenty of people trying to innovate within the web browser space. I think the more important question is, why aren't we seeing that? Where are the super cool new browsers with genuinely innovative UIs?
I don't know! The space does seem fairly stagnant. Implementing a protocol or an engine is a massive task, but innovating in ui, features and the ways we browse the web and explore our history and things like that is something that is already achievable by a single hacker or potentially by small groups of people.
I personally think it's partially due to mindset, where we as users believe that fundamental software is given to us as-is and cannot be changed and should not be changed. The opposite of the hacker mindset. This is of course extra true for massive projects like web browsers, but many things are possible by just forking chromium or using electron or CEF, and that's what you're seeing in recent (albeit niche) innovations like poly pane or bonsai.