If you'll allow me to strawman you a bit, I've always found such arguments facile at best. They seem to reason from something akin to this "As mankind has become more advanced, he has grown from being tribalistic and violent to understanding that fear itself is more dangerous than actual external threats"
Yes, but we've never actually reached such nirvana except in our imaginations. Instead, history has shown us all sort of intelligent and evolved populations both being the perpetrators and victims of indiscriminate violence. In addition, looking at how humans develop and what we know of species in general on the planet, there's never any reason to be afraid of anything -- until there is. That's the problem with induction and not knowing enough about the greater universe. Things are always fine and dandy for the turkey until Thanksgiving comes.
It's extremely problematic to speculate about alien beings. I note, however, that we tend to think in terms of aliens having some sort of policy towards humans: they want to eat us, they want to conquer us, and so on.
Assuming we are just a few rungs up the evolutionary ladder from pond scum, it's much more likely that we're not important at all -- and that if there were some terrible interaction with aliens, it would probably be akin to a man accidentally kicking over an anthill on his way to work than some sort of dramatic invasion or intervention. The scariest thing about potential technologically superior alien beings isn't that they are hostile. It's that they're completely apathetic and have better things to do. Which takes care of Fermi's Paradox quite nicely.