Very generally I think spying happens because nations and non-state groups have interests that are often not aligned with each other, and nations protect their own interests by trying to understand the interests of other nations. Since very few nations are totally up-front about what their interest, goals and behind the scenes actions are, intelligence gathering becomes necessary.

In a perfect world, this wouldn't be necessary. Not only would countries be totally open with each other, but everyone would work together to further the best interests of all. Since we don't live in that kind of world, everyone gathers intelligence on everyone else.

I think it's worth mentioning that people have made the (reasonable, IMO) claim that spying in general actually makes the world a safer place since it removes a great deal of uncertainty in international relations. If countries had to deal with the rest of the world as a black box with uncertain motivations and potentially hidden actions, I think it's reasonable to suggest that mutual suspicion would significantly impede international cooperation.

The core basic legitimate reason is to verify what is said with what is actually happening; to discover the genuine intentions of the opposition. So, olde days, USSR says "we have no intention of starting WW3", and US spying would seek to verify that. If of both sides can convince themselves that the opposition is being honest, diplomacy can work. If you cant be sure of what is being said, you quickly end up in trouble. Both sides legitimately do this as much as they can. Its a confidence builder. It is why both the US and USSR tolerated spies and had a no assassination rule. They both new, in the olde days, that it was beneficial to both to allow each other to spy and gain confidence in each others position.

Trouble comes when nation states extend that to seek political or commercial advantage, or apply it to their own citizens and / allies, who expect to get the benefit of the doubt. Worse still when law enforcement and spy agencies start to combine. This is why I assert that the US, and UK governments treat their citizens as enemies. We have seen plenty of times what its like when intelligence agencies and secret services (SS) are turned on the population.

Unless Obama and Camoron do something now to reign this in, some very dark days are ahead of us.

My guess is that economic and industrial espionage is a big part of it.

alan_cx has the best explanation that I've read on HN yet (though he's wrong about the "allies" part). It's essentially "trust, but verify". You might trust a business partner to not screw you, but you should still read the contracts you sign with them. That's why we tell you to "get it in writing".

Unfortunately, that kind of verification works because there's a legal system in place that you can invoke for punishment of contract-breaking, or at least compensation. At the nation-state level, that verification mechanism breaks down for several reasons. (1) There is no higher legal power to appeal to. The UN is a deal-broker, but it doesn't have the weight to throw around. (2) Things like the Great Firewall and propaganda will naturally make official news sources untrustworthy. You can see some of this on the stock market: most shareholders only have press releases and news reports to decide how the company is doing. But this is only half the story, which is why insider trading happens at all.

So how do you get around these problems? By finding out directly in ways that are harder to fake. Which is what spying is all about.