One example: design. I still haven't seen a single page from Google that looked stunning, or even good (to my eyes, obviously). Their entire design philosophy seems to be infinite-amount-of-monkeys-hacking-css and A/B-testing. They even manage to make their main landing page, with only a handful of items, look quite messy.
The method may be scientific and all, but it doesn't produce good looks.
Any change that deviates too much from the current state of things has a huge amount of friction working against it. Whatever you think of the current design, I defy anyone to come up with a major overhaul that makes up for the loss of warm fuzzies the average user associates with the current home page.
That being said, being ugly can of course be profitable. If you want to signal that you are cheap, for example, the ugliness helps. Just look at the difference between, say, Walmart and an Apple store. So maybe Google wants to look cheap - but I for one wish they didn't.
That's just crap. You can test anything, including entirely different versions of an app. The idea that you can't test good design is what designers tell themselves to justify their operating on gut feeling and pulling opinions out of their asses.
There isn't anything a user does that you can't come up with a way to test, designers are just afraid of actual data showing them they're full of shit. They don't want their ideas challenged democratically by users, they want their own ideas used instead because of some misplaced sense of taste. Engineers do it right, test, iterate, improve, follow the data wherever it leads regardless of your own ego.
You can only test a very, very limited number of designs, out of all the pretty much infinitely many possible ones. Someone still has to come up with the designs to test.
And it is very unlikely that you can go from one design to a better one in incremental, A/B-tested steps. It is just as in language: you can test two different novels, for example, but pretty much all the "intermediate novels" you can think of just don't make any sense.
Yet it's still testable. There isn't anything you can't test somehow and there isn't any design any designer could come up with that can't benefit from incremental improvement as a result of testing.
> And it is very unlikely that you can go from one design to a better one in incremental, A/B-tested steps.
That's your own strawman, no one says you have to. Test radically different designs against each other, testing will still declare a winner and that winner can be incrementally improved by further testing.