Put another way, society and culture are things that are essentially imaginary. They take place in the mind, and I doubt society or culture could organize itself without some shared imaginary space strewn with personifications and myths. Take that away and society and culture become impossible.
Your unease is more warranted for things like evolution, which does not depend on a shared imaginary space.