That’s not how PoS works.
But your main argument is wrong: validators do not control the network. Users running node software do. Running a node is free. If a single validator began to exert unwanted control, the honest majority of non-rich users could slash or evict them.