> But where does the prejudice of "designers only care about visuals" come from?
This isn't prejudice, it's just observed factual reality in the world. The entire flat UI thing came about as a fashion, and had as its consequences the removal of affordances (scroll bars, buttons that were clearly buttons, etc.) You might not like that people question it, but it's nevertheless a thing that happened and is still happening.