Belief based in empirical evidence is at least bound to that evidence.
Belief unmoored from empiricism --- blind faith or revelation --- suffers no such limits. Your criticism would apply all the more so to same, with the further failing that others' beliefs are "wrong" because they ... are simply revealed or adopted faiths. Which is frequently observed.
If my evidence-based belief differs from your evidence-based belief, then we can compare evidence. Evidence does not have to be directly experienced (though for various reasons of psychological evolution we tend often to more highly weight that which is); we can also rely on evidence presented by other trustworthy witnesses. There are numerous social institutions dedicated to precisely this (science, courts, journalism, politics, ...)