"We can't also forget all the craziness in the theistic religions which cause people to do worse things than coordinating over silly ideas."
Spirituality is not a 'silly idea', more importantly it doesn't cause us to do crazy-bad things in general.
'Silly ideas' are things like Imperialism, Communism and Fascism - all of which are secular ideals and have caused massive, worldwide violence to the tune of many massive wars (most wars in fact) to the tune of 100's of millions dead in the last century alone, and continue to hang the existential nuclear threat over our heads.