The last time I tried browsing The Economist web site with lynx, it refused to work unless I changed the user agent string. Gibberish was OK, but apparently lynx wasn't.

Some servers feed different pages depending on whether they think a request is coming from a browser or a bot based on user agent string. Sure it's easy enough for a bot to pretend to be anything, but some servers are still set up to consider the user agent string.

I've set up an extension that randomized my user agent string, to see what would happen, and some major sites were severely broken. Some gave me degraded mode (google did that several times). Also many sites use user agent for things like OS detection, so if you want correct downloads, must have at least partially correct UA.

One can argue that yes, in 2018, there should be APIs that allow to detect all this stuff in a much better way than parsing random mess of legacy markers that is a common user agent. But in reality, parsing UA string is still the case and unfortunately keeps being the case, including very major and technically advanced sites.

nope, user agent sniffing is alive and well. Google is especially guilty, but lots of sites do it. Try changing your user agent and see how many things break.

I've definitely tried browsing with a random User agent - many, many sites are broken. Give it a go!