Not just "really nice". It must be mandatory to respect it.

The only hope I still have is for some kind of fully local LLM-driven "agent" browser that does the browsing for me, navigating search engines, cookie banners and showing me what it found, nothing else.

Unfortunately entire businesses are built around preventing people from using bots, for obvious reasons, so the only obvious way forward to make browsing the web a better experience will also mean ending up on the wrong side of that battle.