"... often judged SOLELY..."
This makes it sound like they can't be judged on anything else, still a very bad way to describe what UX designers do
He's not saying they can't be or shouldn't be judged on anything else. He's saying that, in practice, the person they're reporting to judges their work solely based on how it looks.
It doesn't matter whether or not this is true anyway. It's just a made up example to illustrate the idea.