For PoW, you'd have to know the hash of the start of the chain (the "genesis block") in advance to verify you downloaded the correct chain. That's true, but this hash doesn't change during operation. You could get that hash from a history book if you will.
For PoS, the hash is from the end of the chain and therefore constantly changing. This means the challenge of finding out whether the hash is the right one is a lot more real than in the PoW case, because there is no "common knowledge" to go by which hash is right.