Are you talking about judging his contributions and significance to architecture or his worth as a person?
I get that the two things have some overlap and aren't cleanly divisible. What we do is a major part of who we are.
But I mean, it's one thing to say "I think it [his architecture] is awful", or even "I think his architecture had a negative impact on people/society/cherished values/whatever" but quite another to say "I think he sought the ruination of everything good and decent because he was a demented and feeble mind." Because, yes, I do think the last one would only be appropriate if you actually knew something about the guy...
Edit: And yeah, of course, I'm not trying to censor anyone's opinions. Of course you can _have_ the opinion, you can even express it. I just think that it's not what engaging in productive/civil discourse looks like and, depending on the venue, people may call that out or whatever.