142 karma · joined December 1, 2011
Let me know what country you can start an online service in that doesn't also have a foreign intelligence agency if that's your ethical standard.
Determining if a "selector" is tied to a U.S. person is actually a very subtle and very hard problem.
Let's take a phone number +1 (212) 555-1234
Is this a US selector? It's a selector for a phone in the US, but that's not the same thing as a phone number tied to a US citizen. Let's say I'm following a senior North African pirate with a Maltese mobile +356 2010-1234 and he calls/is called by my number above?
- Should I follow it? Or is it absolutely off limits for me because it happens to be a U.S. number?
- How do I determine if it's tied to a U.S. person?
- What if it's a shared number between a group of associates, all of whom are not U.S. persons except for one?
- Is that number off limits now?
- If it is a U.S. person what should I do with it?
- Pretend it doesn't exist? Turn it over to U.S. Federal law enforcement? Who should I turn it over to? DEA? FBI? ATF? DHS? The Coast Guard? U.S. Customs and Border Patrol?
It's actually a significant intelligence task to figure this out.
What would your collection proposal be then?
You can't determine data of interest until you have sufficient data to determine if it's of interest in the first place. Even the NSA doesn't have an Oracle computer that can look into the future and figure out what vanishingly small percentage of communications are just the ones they should be interested in. If they did, they could also solve the halting problem and rewrite the history of computer science and time-travel.
Remember, the standard the NSA is held up to is that it should not only be following known bad people who are/might be doing bad things, but to ferret out the unknown bad people. Every time there's a Boston or similar, everybody goes and climbs all over the NSA for "not knowing about these guys"
You can't do that until you have a sufficiently large enough collection of unevaluated data to start looking through.
I'm not saying it's right or wrong, only that it's the reality of the task spy agencies have before them.
Right. Most nefarious actors just buy most of their stuff retail. Figuring that out, from whom, what was purchased etc. and if it violates some sanctions regime is one of the things the NSA might participate in, handing off the findings to the State Department of the Commerce Department or whoever for action. The action being that participating companies would not be allowed to do business in the U.S. and would be monitored for further sanctions violations in case other U.S. involved companies want to do business with the retail outfit that sold the whiskey. No big deal if you're Mom & Pop liquor store in Italy with no U.S. involvement. But who knows? It could mean that your supplier is a U.S. based company and may not be allowed to supply you any more in the future.
In practice nobody gives a shit about a few cases of whiskey, but the same practice might be used for large shipments or for items that are undesirable for North Korea to obtain.
> Really? why should anyone trust in anything coming from the NSA when you are systematically lying again and again? why should we listen to anything you say when historically, part of your strategy is to try to influence the PoV of society || specifics groups?
OP is not the NSA. OP was an employee for a couple years and can only comment about what he specifically knows about, just like anybody. Take what he says with that context.
alfiejohn_: https://news.ycombinator.com/item?id=6911106
(ex-)Employees can write about anything, but if it involves IC related topics, they have to submit to a review before publication. All those various books about life in the Spy world or Special Forces etc. all went through similar reviews.
malloreon: https://news.ycombinator.com/item?id=6911226
> There are articles suggesting this is happening many thousands of times per year - shouldn't each of these 'regular employees' be put on trial? They have committed serious crimes.
Quite often they are. The minimum penalty is loss of clearance and job (which is a career death sentence in the IC), and depending on what they did and if the agency feels like it's a good expenditure of resources, some kind of criminal prosecution may be involved.
e.g.
- Employee sees if his own cell phone number has been collected (violates rules regarding looking up U.S. persons): probably a firing, but not worth criminal prosecution
- Employee starts a side business and starts monitoring communications of his competitors to get a leg up, probably a firing + prison time.
te_chris: https://news.ycombinator.com/item?id=6910858
> Thank you so much, kind American intelligence guy, for having the grace to not look at USA citizens emails, all the while not even mentioning foreigners, who should apparently just lie down and take it.
Foreigners don't just "take it", they have their own nation's intelligence apparatus collecting against their foreigners. Let me know when there are mass protests against foreign collection in the rest of the world and maybe then your complaint will make some kind of sense. Until then it's just bitter complaining about something you knew all along anyway. It's kind of the point of Intelligence.
lucb1e: https://news.ycombinator.com/item?id=6911262
> Why is a distinction made between US and non-US people?
Because, at least in the U.S., there's a distinction between IC and LE entities -- this is not a distinction shared by most countries. LE deals with domestic events and U.S. people, while IC deals with non-domestic actors and locations. There's a bit of overlap, for example, a U.S. citizen doing bad things in a foreign country gets a bit of both, or a foreign national in the U.S. But that's generally where the line is drawn. U.S. law doesn't apply outside of the U.S. and it shouldn't. Can you imagine the world if the U.S. suddenly decided that France had to apply by all U.S. laws including the U.S. Constitution? If people think New Zealand cooperating on
> What makes me a potential criminal, and Mr. Smith not?
The reverse is true of U.S. citizens w/r to other country's intelligence apparatus as well.
IC agencies generally aren't looking for criminals...meaning they aren't looking for people who broke U.S. law. They're basically looking for threats to U.S. soil, citizens and concerns. LE agencies look for law breakers. That's one of the reasons why the trials at GTMO are such a mess, guys were wrapped up and sent off to Cuba for interrogation based on "Intelligence" but not using information collected as part of a criminal prosecution. IC and LE really do operate under very different methods (at least in the U.S.).
princeverma: https://news.ycombinator.com/item?id=6911007
> I am a foreign national, I and my company uses services provided by a US company (email etc.), and this gives right to you guys to collect and ready my emails?
Yes. Look at your question from a different angle, under what legal regime is the U.S. not allowed to do so? Or the contrapositive, what are you doing to prevent your country from spying on U.S. Citizens?
Tarang: https://news.ycombinator.com/item?id=6910916
> If an employee had a contrarian opinion to the NSA would it be declassified like this one?
Yes. You can check Amazon yourself for books by former NSA employees who don't side with the Agency. Their books were all reviewed by the NSA for classified information, but not for dissenting opinions.
muglug: https://news.ycombinator.com/item?id=6910838
> Do you think Snowdon's revelations had any beneficial impact, or is your view of them entirely negative?
Disappointingly no. At least in my circles, Snowden is not mentioned in polite terms. The likely outcome will be no change in NSA collections or capabilities, but automation in system administration, better collection of auditable information on IC employees and more compartmentalizing of the activities even further.
room271: https://news.ycombinator.com/item?id=6910827
> My question to the OP: even if you believe that at the moment abuses are rare and that your colleagues are trustworthy and law-abiding, does the capability and level of information concern you in terms of the potential for future abuse it enables?
I'll toss in my answer too. I think it is serious cause for concern. But the kind of work the IC does is hard and I'd say impossible to get right. Anybody who thinks the kinds of work that most of us do in our day-to-day businesses is hard have no idea how really difficult the industry is. In a normal business, if you get things wrong you might get sued or shut down. In the Spy industry? countries might fall, nukes might loose and people might die. Doing that kind of work in balance with people's rights, or at least not negatively impacting truly innocent people is among the hardest things to do in the world.
aaron695: https://news.ycombinator.com/item?id=6911046
> "All that is necessary for the triumph of evil is that good men do nothing."
Doing nothing would be for the U.S. to not spy at all, which is a sentiment shared by many in this forum. And then bad things would certainly happen for the U.S. because no other country would do the same. Sometimes doing the right thing really involves doing hard things that don't feel good. I won't make any apologies for it and nor should the OP. The kind of work that the IC and the military does, make no mistake about it, is ugly ugly work. It is very important though that this public debate be held often, and constantly. The IC/Military construct is like a huge guard dog. You want it a bit mean and angry, but you don't want it to bite you, the mailman or the neighborhood kids...it's not so bad if it goes after the occasional solicitor or burgler. So it's very important that you keep a constant eye on it and keep it in check. Because it will dig up your yard, chew up the sofa and otherwise misbehave.
> Anyone who defends the NSA on the grounds that it only targets those who are worthy of targeting needs to convince me that another COINTELPRO will never happen. I would actually welcome such an argument, since it would make me feel a whole lot better about this.
You raise a very good point. I don't think there are many in the IC who would have a problem with more (and better) oversight...it helps them feel more legitimacy in what they're doing. All that being said, assume that there will be another COINTELPRO or similar. It sucks, but to be perfectly honest with you, you aren't important enough to convince one way or the other about it or about approving of the various collection programs the NSA is running. If half of the comments here are to be taken at face value, almost nobody on HN knows enough about the issues involved to be consulted or for their opinion to be considered. I'll draw a parallel here to the outcry over technology centered court cases or software patents, an opinion often given here is that engineers or software experts should be involved in deciding those cases because the layman doesn't know enough about it to have an informed opinion. It's just as true with IC issues. What's troubling is that oversight is via Congress, who until the last election were likely laypeople themselves. So the expertise to properly evaluate, oversee and monitor what's going on simply isn't there.
sedev: https://news.ycombinator.com/item?id=6911078
> This reads like it was penned by someone who's never heard of the Stanford Prison experiment or Milgram's research...
You bring up good points. One of the issues with the IC is one of internal monitoring (as we're all now seeing). Who watches the watchers? There's notionally a number of very serious laws, or committees and counter-X professionals who are supposed to be doing this, but to be honest. Once you're cleared and stuck on a project, there's really very little day-to-day oversight of any kind. I personally would welcome a higher level of scrutiny, but I think the equation of "spend manpower watching watchers" vs. "spend manpower looking for bad guys out in the world" has, and will continue to, balance on the later.
https://news.ycombinator.com/item?id=6911091
> If you'd never heard of parallel construction before today, that seems to powerfully undermine your credibility.
He only worked there for two years on a single program. I challenge anybody here who works for any organization larger than 3 people to have perfect knowledge of every single thing their organization is doing. Unless you work on the bits that interface with the LE community you'd probably never have had opportunity to know anything about it. I bet he also doesn't know the details of companies the NSA contracts to do their plumbing or handle their trash.
EthanHeilman: https://news.ycombinator.com/item?id=6910896
> The NSA has a history of sharing intelligence with LE, to state that the NSA is not a LE agency is extremely misleading, if not an outright lie.
I don't think you understand what a LE entity is, the powers and limitations that LE organizations operate under or how they differ from IC organizations. Saying the NSA is LE because they cooperate with LE is like saying the Department of Agriculture or Labor is an LE agency because they sometimes have to cooperate with LE.
jjoonathan: https://news.ycombinator.com/item?id=6910922
> It's good to hear that many NSA employees take the police/military distinction seriously, but we know for a fact that some higher-ups don't...
This is true, and it's particularly vexing to work in the IC and have senior officials asking you to participate in outright illegal activities. It doesn't happen often, but I've seen both people ruin their careers saying no and those that want to keep their job and say yes. It's usually under the auspices of "helping out to stop bad things", and it's very hard to say no when activities are couched that way...despite very many of the people on here talking publicly about their very righteous and moral high ground, placed in the same position, the vast majority of HN users would want to stop bad things from happening even if it meant crossing the line a bit here or there. Because, honestly, if you choice is help stop innocents from getting killed vs. sit on your hands because of some futzy law someplace that ties your hands, most of us would feel like we'd rather take the immediate action to stop the bad guys. "Mission Expediency" is the word of the day when it comes to these matters.
mtgentry: https://news.ycombinator.com/item?id=6910911
> No offense to OP, but this reads like propaganda to me.
I can confirm that this represents the mindset of most of the people I've worked with in the IC. It does take a certain kind of personality to sign-up and go through all the hassles involved with getting employed in the IC and that self-selection seems to attract a certain kind of personality type.
> But I wouldn't be surprised if there was some sort of concerted effort by the NSA to encourage a dialogue with hackers on platforms like HN.
There really isn't, other than to not discuss classified information. Keeping track of what's classified and not when you spend near a third of your life and half of your waking hours dealing with only classified things is complex enough that most people just don't engage with the "unclean" public.
jurjenh: https://news.ycombinator.com/item?id=6911235
> What I'd be more interested in is how much this issue is being discussed internally. If these discussions are allowed, or even surreptitiously encouraged, then I'd take that as a possible internal propaganda push, subtle as it may be.
Discussions about these issues, in the way that they're being discussed here, are not terribly common. The milieu of working in the IC just doesn't lend itself to these kinds of topics, in these kinds of ways, as focuses of conversation. Mostly what's discussed is the lines you aren't supposed to cross and the penalties for crossing them. But more often then not, casual conversation is about anything other than IC topics. After a hard day of spying, you really want to just engage in something else.
Bizarrely, discussing Snowden might be tricky inside of the IC because you might discuss something that was leaked that you technically don't have a need-to-know for. So specific cases like this are not often serious topics of conversation.
(continued next comment)
There's lots of condemnation of the poster, and the NSA practices and some of the murkier parts of this article. I thought I'd tip in with some explanations as possible while staying outside of anything classified or naughty.
jonknee: https://news.ycombinator.com/item?id=6910978
- "It's compartmentalized enough that the individual actors can justify their actions by the assumed competence and benevolence of the others."
It's compartmentalized a bit more than the OP lets on for mostly security/separation of concerns/need-to-know reasons. For example, a Air Force analyst who is cleared to view TS//SI material won't have access to the NSA systems directly. Some of the NSA systems have external (Intelligence Community (IC)) facing equivalents that omit quite a bit of the information that less scrutinized IC analysts shouldn't have access to. w/r to the information the NSA collects, NSA employees and contractors are held to stricter standards about how that material is used and treated. An analogy, a minor commits a crime and his record is sealed. The local court employees who handle the record, the judge etc. have really nothing that prevents them from leaking that information to an overzealous cop or lawyer or some such other than the standard to which their held for their job. It's more or less the same thing with the NSA.
> The mental leap here is subtle, but substantial. Since I have been told I can't use US selectors , I assume the system enforces this.
Actually, one of the higher standards the NSA employees are held to, and I believe they sign something to effect is that it's outright illegal for them to do so and even one misuse could result in loss of employment, clearance (a death sentence in IC heavy employment areas) and possibly time in prison as a felon. This is taken very seriously and I've never known an NSA employee to not treat this rule and US citizen data as radioactive to them.
https://news.ycombinator.com/item?id=6911054
> Definitely a bizarre mix, I thought it was a parody a couple of times. To combat the threat of nuclear war with the completely isolated totalitarian state of North Korea we must create and store copies of all global communication...
It's easy to generalize, and if the world worked as simply as the model you propose here, then things would be much better for everybody, but it simply doesn't. For example, to uphold various sanctions regimes, by law, the U.S. must know if a business has connections two hops out that are linked to any bad activity. For example, how did Kim Jong Il buy all his whiskey? It's outright illegal for a U.S. company to sell to the North Korean government. Okay, so they sell to an overseas distributor who then sells to the North Korean government. Turns out that's illegal as well and the government must take action to not allow the U.S. whiskey maker or the distributor to operate in the U.S. any longer. Okay, so the whiskey make checks out their distributors finds one who doesn't sell to NK, but one of their customers does. Same deal, it's illegal for anybody in that chain to operate in the U.S. After that, the chain becomes so long it's not worth looking into and Kim Jong Il was eventually able to get his whiskey.
Just talking whiskey and North Korea here, but you can guess it goes for all kinds of goods and countries under various sanction regimes. So how do you propose things should be collected? Collecting only on North Korea gets you nowhere, it's everybody else who may or may not be supplying whiskey to the Norks that makes things much harder and requires a much larger collection apparatus.
https://news.ycombinator.com/item?id=6911216
> It's helping diplomats illegally snoop on our allies.
Good! Our allies are most definitely snooping on us! Spying and espionage is sometimes called the second oldest profession for a reason. There's been no time in history that two countries aren't doing a bit of spying on each other, most especially at the diplomatic level.
rst: https://news.ycombinator.com/item?id=6911150
> In fact, it's been known for months that the DEA receives intercepts from the NSA in such volume that they have an office devoted to handling them (the DEA's "Special Operations Division").
This is a problem. In general, the work the IC does in collection does not hold up to LE scrutiny. Having worked on both sides of the fence, LE is both more difficult in some cases and easier in others to work in. For example, you need a warrant to gather phone records in LE, but you can share those records more freely once you have them. In the IC the opposite is true, you can pretty much get whatever you need, but it's virtually useless if a criminal approach is taken. That's why it's often simpler to blow up the target then to arrest and try them. Parallel Construction is an investigative focusing approach that saves LE from getting collection warrants that go nowhere. The IC approach is to find the connections or whatever, then help LE figure out where to focus their warrant-based approach in doing the same collection from their side. Scrubbing U.S. Persons IC data and reusing it directly for LE is highly illegal for all of the participants involved.
revelation: https://news.ycombinator.com/item?id=6911022
> Well, following his explanations, you can fail the polygraph and just do it again. The cost of failure is zero, so really just keep trying.
Actually the penalty after enough tries is no clearance which means no job and a permanent record that you were denied a clearance...which pretty much deep sixes any attempt in the future to get one. In some parts of the country, like the Washington D.C. area, that's virtually a career death sentence.
kabdib: https://news.ycombinator.com/item?id=6910969
> My best friend's dad was a spy in the CIA
> During the 70s and 80s my dad worked with Russian scientists
> So, how likely is it that my email is read, that my phone records are looked at, and so on? What are the chances that I'll have trouble the next time I cross a border or try to board a plane? One percent? Fifty percent?
Assume it is collected but probably not read, but not for the reasons you gave above. There's just simply not enough manpower to read everybody's email, and it's a useless thing to try to accomplish. Now suppose one of the guys you email also emails somebody who's "nefarious" in some way. Then yeah, maybe your email is read. And if all you talk about in your emails are things that don't involve an armed insurrection against the United States you'll probably be filed into the "don't give a shit" bucket and the analyst will move on.
A common thread here is that everybody who's worried about their email being read seems to assume that whatever they're doing is important enough for it to get read. Trust me, it isn't.
(continued next comment)
So in 99% of the cases it's not like you make a decision to "classify" something. It is by its nature classified.
Think of it this way. You have a friend that tells you via email a secret. This secret can only be known between you and your friend. For example, you are arranging a surprise party for a third friend.
Using this knowledge it is your job to get a birthday cake that has the person's name and age (and a decoration) on it. But the only baker in town is also friends with the birthday friend. You don't want to risk the baker leaking the surprise of the cake to your friend. So you decide to buy a blank cake, and decorate it yourself.
This cake is also classified at the same level of the original information (Classified//only between you and your friend). Because it is derived from the knowledge of the birthday party email).
Now your friend prints off decorations with some party specific information, say a sign or some streamers or something. Those are also classified at the same level.
Suppose there are some other odds and ends that are a result of this email and the party plans. Say a "making of video" for the party etc.
Now let's say you also hire a clown for the party. You only provide the clown the time and place to come and perform, not any other details. Because the clown isn't in on the original secret, the arrangements for the clown and the various transaction documents pertaining to it are at a lower classification level. You might still not want the birthday friend to know that somebody he knows hired a clown on his birthday (a convenient coincidence) so you swear the clown to secrecy.
Even if it gets out and the birthday friend asks the clown directly, the clown has plausible dependability and doesn't know why he was hired, only that he was. So it's not good that your friend finds out, but it doesn't entirely screw up the surprise.
I'm stretching the analogy a bit, but hopefully that makes it clear how knowledge of single piece of information (the email that established the surprise party) can end up producing so many classified documents and how that process happens.
http://www.dss.mil/isp/fac_clear/download_nispom.html
Army Grunts are required to have a clearance. General Infantry (18yo cannon fodder in your parlance) must have at least a Secret clearance. Army Cooks, may even be required to be cleared.
Think you can run around with 5th generation night vision goggles and state of the art body armor, talk on encrypted radio equipment, know troop movement information, chase after specific enemies and report back on activities on Secret level systems without a clearance? Think again.
Let's do a thought experiment. Let's say we can say there's a piece of information that is so dangerous, so important, that releasing that information could endanger the entire population of the planet. Say, the nuclear codes.
These are arguably the most sensitive information in the world. Why would there need to be a classification level above and beyond TS and the appropriate compartments and or SAP program to protect them? Some sort of Super Top Secret?
Or for fun, let's say Area-51 has alien tech there, or we have a secret military base on the moon, or a Stargate. Why wouldn't it just be protected under a SAP? A Super Top Secret doesn't buy you anything at all in terms of protecting that information. Nor does a Super Duper Top Secret. As somebody not in the SAP and not with a need to know, I wouldn't even know the SAP exists to protect the Stargate program, or the alien autopsy videos, or whatever. Only the people in the SAP and a handful of people managing the SAPs of the agency that created it even know of the SAPs existence and/or what it protects. SAPs can even have sub-compartments that provide even further protection.
Practically speaking this offers what can be perceived to be "high levels of clearance" but in fact are all just plain jane Top Secret.
You can get a clearance at 18, and people live into their 70s and 80s pretty regularly. The 4.2 million with a clearance are not the same ones that had a clearance 10, 20, 30, 40, 50, 60 etc. years ago.
We had a draft until the early 70s. Tens of Millions of people have been through the military system, and this doesn't even count civilian types, contractors, cleaning people, building facility people, secretaries, etc.
The Vietnam war saw about 9 million people in the military for example.
There's about 25 million vets in the U.S. right now.
It's pretty simple math to arrive at number well above the current number. Remember, just because you aren't in the club, doesn't mean it isn't big.
Let's look at what 10% means. There are ~300 million people in the U.S. 10% of that number is 30 million. Let's say that every single man and woman who is currently in the military or has previous served has a clearance (with no overlap). There are about 3 million active and reserve people right now in the military.
25 + 3 = 28 million. Now let's add in non-military government civilians with clearances, the CIA for example is a civilian agency with an estimated 20,000 people, DIA, 16,500, DTRA, 2,000, DOE: ~110,000, NGA 16,000, NRO, 3,000, NSA 20,000 (civilians only, 38,000 total), DHS, 216,000, DOJ, ~112,000, State, ~50,000, etc.
I'm not even including treasury. And we're up to almost 29 million people.
Now how many contractors do you think have Secret clearances?
Lockheed Martin employes over 100,000 people, Northrup Grumman, 120,000, General Dynamics, 91,200, SAIC, 46,000 etc. etc. etc.
And we rapidly go past 1:10.
Now obviously not every single person I've listed has (or has had) a Secret clearance. But until you start shaving off huge percentages out of each organization you're still between 1:10 and 1:15.
If you disagree with these numbers, the onus is on your to provide better ones.
Quite often it could be that the investigator could not ascertain the relationships on the wife's side and/or they don't have a citizenship yet.
"Well I have a clearance and can tell you this is how it is" is probably the mark of somebody who just wants to win the debate but doesn't know squat.
I've worked on SAP programs that had 3 people on them. Some of the larger compartments probably have hundreds of thousands read into them.
Also, I work with plenty of people with TS/SCIs who have used drugs in the past.
The key is just being honest with the investigators.
You are right that people with too much debt, gambling problems, current drug use, etc. don't get one. But that makes sense. I've worked with some pretty sketchy characters though and they didn't have a problem getting or keeping there clearances.
I don't think I've ever personally known somebody who had theirs denied, I've known one person who had their taken from them for doing some questionable things.
"You said that there is no level strictly above Top Secret (just comparmentalizations within that). Is there any actual evidence for that? (I mean, non-classified evidence; I'm not asking you for classified information.)"
There are no levels strictly above Top Secret. At least as far as the military is concerned. SAP programs and compartmentalizations provide more than enough OPSEC to keep things classified. For example, I don't personally have access to the Nuke Codes. But the guys who do all have Top Secret clearances and are read onto some specific compartments and SAPs.
There are also clearance equivalencies at other agencies. DOE for example, has a different kind of system, but they more or less map to DoD clearances. Some agencies are highly compartmentalized, the CIA for example.
"Where did you get the statistic about 1 in 50 to 70 Americans have had at least a Secret classification? That seems absurd."
I don't know the exact number, but I have a pretty good idea that agrees with some back-of-the-brown-paper-bag calculations, it's perfectly within reason.
For example, say there are 1 million people in the military right now. And say all of them have at least a Secret clearance (it's probably more like 70%-80%, most jobs get you at least this clearance level just as a matter of course). That's about 1:300 (or 1:430 or so depending), say right now there are about 25 million veterans (from the Census), that's already 1 in 12.
During the Manning news, it was reported that currently there are more than 4 million people with clearances and one million of them had Top Secret clearances.
http://www.dailymail.co.uk/news/article-2041326/More-4-MILLI...
That pegs it at 1:75 right now with Secret levels and 1:300 with Top Secrets.
1:50-1:70 for people who do or have had access to classified material is is extremely conservative.
At TS it starts to get more restricted. You have more ability to request access to things outside your general area, but you may/may not get approved.
In the Compartments, my experience has been they are surprisingly global in nature, but focused in content. Meaning I can pretty much get that "kind" of information for any place on the planet, but it's only that "kind" of information. There's some exceptions, but not many.
SAP programs tend to be where it gets very focused. Often because they represent information only collected in your area using only a specific means. So there isn't a global collection of that kind of data -- well, there might be, but you aren't in all of the SAP programs to try and figure it out.
There are many many networks, agency specific ones, site specific ones, standalone networks of half a dozen computers (say for a SAP program), sometimes some that give access to different compartments, etc.
Even at the Secret level, you have access to a mind-boggling amount of information. There's nothing particularly surprising at the amount of information Manning had access to. I'm surprised he didn't make off with more to be honest.
I've also found a couple things to be true about moving up in classifications for those that are interested:
1) The higher up you go, the lower tech things tend to get. The IT systems on super duper squirrel SAP programs are positively ancient. There's no super secret A.I. computer that only the President and a strangely well groomed computer super nerd know about. It's probably a 10 year old desktop with a slightly broken CRT monitor.
2) There are some really cool toys at high classifications -- neat space cameras and all that. But the vast majority of the really neato stuff is at the Secret level. All the tanks and bombs and military toys are pretty much at that level. "The Army Fights off of SIPR (the secret network)" is the mantra. If you go to Wikipedia and start looking around at various pieces of military equipment, you'll probably seen 95% of the toys that anybody with a clearance has authorization to know about.
3) This piece is absolutely true, and the voices that speak in loud tones about vast conspiracies but who've never been in this world are perfect examples of what this article is talking about. Once in, the utter mundane nature of most of it is almost overwhelming.
4) An amazing, astonishing, amount of information is available completely for free on the Internet and in Libraries and other completely open places. Often the best information comes from there. There's even a field called "Open Source Intelligence" that requires no clearances at all! It's basically internet surfing and report writing to answer specific questions like "does the Prime Minister of Japan have a mistress?" or "what kind of space launch capability does the Ukraine currently possess?" or "what's the phone number of this Falafel joint in Paris?". Some of the best, most insightful, and most comprehensive stuff I've ever seen was put together with access to the Internet, MS-Word, Google Earth/Maps, Powerpoint and a Library Card.
Here's a great example: http://www.nkeconwatch.com/north-korea-uncovered-google-eart...
5) Physics applies even to the Intelligence Community.
If you have a decent education in Engineering or Physics, you probably already are capable of knowing or understanding 99% of the capabilities of the highest of the high speed super spy systems. If Physics doesn't allow it, there's no getting around it. That doesn't mean people haven't engineering some clever things, but we're still limited by things like the speed of light, or available materials that can defract or focus it (none of this "enhance, now focus on the reflection, enhance, now that reflection, enhance, rotate, enhance!"). Radio waves can't penetrate the earth, low frequency waves that can penetrate lots of stuff still propagate to the inverse square law. Encryption can be broken, but it takes lots of computation power to do it often enough -- being able to break a single message in 12 months with a top 100 supercomputer doesn't mean squat if you need to break a million messages a day.
http://en.wikipedia.org/wiki/Radio_propagation
6) Almost everything I've said here is on Wikipedia, or linked to by Wikipedia. Anybody really interested in this stuff, and determined enough to click link a few dozen times could figure this out. It's often surprising how passionate and conspiratorial some of the most uninformed comments are, even here on HN where there's no excuse to not have done just that.
7) Getting a clearance is really not that big a deal, but there can be a catch-22. Most jobs that require a clearance won't bother letting you apply for one. But there are still plenty of places out there that do. Lots of R&D organizations will do it. Non-profits that contract with the Government will often do it. It's easier to start young, the investigation is faster. But the reason most don't is that they likely have nothing for you to do without a clearance and you'll just be sitting there eating up overhead budget for months on end while you wait for the process to work its course.
8) Security managers, (FSOs and SSOs) are almost uniformly idiots. If you think Comcast's customer support sucks, try dealing with these clowns. Misfiled paperwork, nonresponsive to emails and phone calls, generally don't know their jobs, etc. If you find one who has half a brain you hold on to them lie you are drowning. The only thing they are good at doing is making sure that people without tenacity are filtered out of the system.
9) This job often sucks. Paperwork you can't believe, endless training lectures, shitty work environments, old IT infrastructure, endless hassles to get to do your work, constant barriers to just getting things done, weirdo people, lots of unmotivated do nothings, no windows, can't talk about work, forget one of a couple dozen passwords, cut off from the outside world most of the time, no idea what the current news is most of the time, pay can often suck compared to private sector, access to data is usually buried in some non-machine readable format.
10) The job can be rewarding. More than once while I was doing work, stepped out for a break and saw some breaking news on CNN or someplace that was reporting on recent evens right where I was! Cool shit. Other times you get to play with tremendous amounts of raw data, if you are a data junky it can rock. Cool toys, things used in ways you probably won't believe, gather and collect that data. And sometime really incredibly interesting people who've been around this stuff for decades. Some of the smartest people I've ever met work in this field, outright 1 in a century geniuses. And savants who've memorized the chemical makeup of every piece of military hardware in North Korea's arsenal, who the current commanders are of all the tank battalions and can draw a spectrograph of all of the elements by hand on a whiteboard.
You can move information down only if the information is at the level you are moving it to and it's been properly signed off.
e.g. something that is marked Secret, but written to a Top Secret system, can be moved back down to the Secret system)
see previous comment here http://news.ycombinator.com/item?id=3298594
Keep in mind that there are probably north of 1 in 150 Americans with current access to classified material. It's a huge number and just because you aren't in the club doesn't mean it's really all that hard to get in. Get a job that gets you a clearance and you can see what's on the other side of the curtain. Believe it or not, it's amazingly mundane.
a) due to how the information was collected
b) something derived (citing) a piece of classified information.
So a special collection platform may collect material at some high level of classification, anything that references that information (it could be 100,000 reports) are also all classified at that level automatically.
It's just how the system works not individual politicians stamping individual pieces of information as classified.
In the U.S. here's how it works (I'm writing this from the perspective of a contractor):
You are hired to work on a government contract, that contract requires you to work on xyz project that requires you to handle information classified at a certain level (or with certain caveats or handling requirements). You fill out a bunch of paperwork (http://en.wikipedia.org/wiki/E-qip) and it's submitted to the government. The information you put down isn't really a whole lot more interesting than what you might put down on a home loan application, but you sign some consent forms that the investigator can do some credit checks, that sort of thing. You also put down some references they can contact.
Depending on the level you are applying for, the investigation may take longer, particularly if they interview your references. The interview questions are usually simple things like, "did you work with so and so at such and such place?" "have you ever heard them talk about overthrowing the government?" that sort of thing.
If you're a normal person, no serious prison record, drug addiction, serious mental health problem, or threatening political viewpoints (card carrying member of the nuke the US party) there's really not a lot that can prevent you from getting a clearance. Even prior drug use doesn't necessarily prevent you from getting one.
A Secret clearance has a very low bar to entry. You can get one after application in perhaps 3-6 weeks. I can't even get a cable guy to come to my house in that length of time.
A Top Secret clearance takes a bit longer, and is slightly more involved, but it's on the order of months to a year.
http://en.wikipedia.org/wiki/Security_clearance
Most of the time people either get a Secret clearance of a Top Secret clearance -- there is no such thing as a "clearance" above Top Secret but people can often be confused by special accesses at those levels (explained below).
Operating at the Secret level, you'll have access to most of the information that is classified at that level. If you've read any of the wikileaks State Department stuff or the Afghan and Iraq war diaries you've seen what kind of stuff it is. Most of the time it's just information that the government would rather not go public with, but isn't really all that interesting in nature. Records of events, meetings, general information reports, troop movements, that sort of thing.
Probably 1 in 150 Americans has at least a Secret level clearance, and probably 1 in 50-70 have had one at one time (there are a lot of people that move through the military and/or for the military).
The Military's information systems are generally geared around the Secret level of classification and done on an Internet-like network called SIPRNET, There's even a Wikipedia analog and a Google search on it. It's like using a slightly shoddy version of the Internet as it was 5-10 years ago.
http://en.wikipedia.org/wiki/SIPRNET
http://en.wikipedia.org/wiki/Intellipedia
To be honest it's not really much more interesting than using your regular run-of-the-mill corporate firewalled intranet, except it's an unusually large organization.
Not all Secret information can be shared with our allies. Why? Well, we may be fighting a war with say, New Zealand at our side, but also investigating a case of attempted bribery where NZ is trying to smuggle sheep into California or some such. We don't share the bribery investigation data for example.
To deal with this we use what are called "handling caveats". Something shareable with say Canada and Great Britain might then be marked as SECRET//REL TO USA, GBR, CAN or similar. There are also group handling codes like NATO, ISAF etc.
http://en.wikipedia.org/wiki/Classified_information_in_the_U...
http://en.wikipedia.org/wiki/International_Security_Assistan...
There are also other classification markings that are used as caveats. They look kinda the same and are called compartments. It's generally just more restrictions on who can see the information.
http://en.wikipedia.org/wiki/Sensitive_Compartmented_Informa...
People use 'SCI' like it's the same as "SECRET" or a some super high level classification. But what it really means is that it is information gathered in some way that we would really really rather keep private and thus you need another level of need-to-know to get access to it. Typically it's this way because billions of dollars was spent getting that information gathering capability and replacing it would be billions more or revealing it could expose people to severe risk, harm or death.. SCI compartments exist at all levels of classification.
Often knowing what the information is showing can directly inform somebody how it was gathered as well. So it's not just the means that's protected directly, but the data as well.
To see this information, you need to be working on a program that requires you to work with information in that compartment (need-to-know). And you will be "indoctrinated" or "read on" into that compartment. Which usually involves filling out some more forms, submitting the application, and watching a boring video telling you what the compartment is all about. There are many compartments. Compartments can also have sub-compartments.
However, in some cases, the information is so super sensitive (almost always meaning that people could be killed if it becomes known) that the government wants to make sure you can be trusted with it. So you might get polygraphed. Usually they just ask you things like "have you ever considered trying to overthrow the U.S. government" and other similar. You might also go through a slightly different polygraph with highly personal questions about your sex habits.
Even more restricted are Special Access Programs (SAPs)
http://en.wikipedia.org/wiki/Special_access_program
These are things like the Nuclear Codes. Even the names of the SAPs are super double probation classified. Often they are one super specific kind of information, and they are managed very closely by the agency that creates them. Very few people will be granted access to the SAP.
All of this holds true for Top Secret clearances and information, except everything is just that much more carefully controlled. You have to have a reason for accessing it, most people don't, and you have to fill out lots of paperwork and have lots of background investigation stuff.
You are encouraged frequently to try and accomplish as much as possible at the lowest classification possible. Mostly so we can share the information (all of which is declassified after 25 years max). But sometimes we just can't and things have to move up in classification.
Everything has a legal hurdle. There's lots of lawyers and other bureaucrats involved in everything. You have to report where you are everyday or establish where you'll be if you'll be out of contact for any length of time or you lose your job. If you did something bad while missing, then you'll probably end up in Federal prison for a very long time.
There's definitely too much stuff classified. What most people who don't know anything about the classified world complain about is this problem. It's actually reasonable to argue this. What they don't realize is it's a pain in the ass to keep stuff classified. What they also don't know is that everything declassifies after 25 years (or sooner) unless it's something super special sensitive (nuclear codes). This is a tremendous pain in the ass for the government to go through, but in the interest of stopping information that should be free from being locked away, it's done and most people I know in the field think this is great. Because keeping this stuff secret is a drag, you can't talk about most of what you do everyday at work with anybody outside of your work. You can't talk about it at home, even if your spouse is cleared.
http://en.wikipedia.org/wiki/Executive_Order_12958
http://en.wikipedia.org/wiki/Executive_Order_13526
http://en.wikipedia.org/wiki/Declassification
To put this in perspective, we're almost halfway to the point where everything about the lead up to the mistake of the Iraq War will automatically become declassified and available via FOIA requests.
All of this is done in special facilities called SCIFs
http://en.wikipedia.org/wiki/Sensitive_Compartmented_Informa...
(technically work that is not SCI protected doesn't occur in a SCIF, but that's what everybody calls them). They usually have all kinds of access controls, some have several layers.
For example (a real one), you may have to pass through a gate with an armed guard, an armed guard at a desk who checks ID, a proximity badge and keypad turnstile, a locked door with badge and keypad, an elevator with badge and keypad, a finger print scanner badge keypad combo, and then a safe to get to your hard drive, which is then protected by an encryption key, the login user/pass for the system, then a user/pass encryption key for the database you are accessing and finally a special decryption password for the file you need to download and unarchive from the database. This doesn't even include all of the signin/out logs and other paperwork required to get through a day.
Depending on your clearance, you may end up with several different computers at your desk at the same time, usually connected by a KVM of some sort. Something like this isn't all that unusual
http://upload.wikimedia.org/wikipedia/commons/6/6f/Intel_Gre...