What I Learned in 40 Years of Doing Intel Analysis for US Foreign Policymakers [pdf]
cia.gov
cia.gov
Some observations of my own, that I can see echoes of in this paper:
Decision-makers have a mental model where analysts' job is to synthesize facts and provide them, and it is the job of the decision-makers themselves to put the facts together into a big picture and/or plan of action. This is generally a poor set-up. Firstly, "facts" with any certainty are thin on the ground, there's only inferences from small spots of data in between clouds of blind spots. Secondly, the analysts have more experience with the trends that they study, and often more training in how to incorporate them into a big picture. Most managers will say "I want all the data," but if you give them what they ask for, they will most often draw incorrect conclusions. You have to spoon-feed them, even though neither party particular wants that.
People don't understand the limits of data, and they sure as heck don't understand uncertainty. If you make 100 predictions, and are 99% sure about all of them, then on average one will be untrue. And all you'll ever hear about was that you were 99% sure and were wrong. In my job, there's actually a consideration about not providing data that we don't totally trust (as opposed to providing with caveats), because a single data point that's wrong erodes trust in data in general. It sounds like that's not an option to the CIA: if they're tasked with a question, they must provide an answer (which is a problem itself).
We've found that if we our client to use our analysis, we actually need to write it with his audience in mind. We literally write it with the design goal that the client be able to copy-paste our words or graphs into his own deliverable. Because that is literally what will happen. The more effort it is for the client to consume our work, the more likely they are to save themselves the time and ignore it. And as bad as that is, it's also probably the best way. If the client re-words things, he may misunderstand things. Encouraging plagiarism cuts down on the game of telephone.
I've spent most of my career in marketing, and as far as I can tell, this happens everywhere. Well, almost everywhere. The one exception I've encountered was working at Apple, where people expected you to hammer out a beautiful and convincing Keynote deck on your own. Everywhere else, people more or less copypasted whatever decks, charts, graphs, and bullets their agencies had sent them.
The problem with copypasting is that it can lead to a game of telephone. A sequential erosion in the quality of information. If you're not keeping track of who's been lifting and plagiarizing and pasting which information from which sources, at how many steps along the value chain, you end up with a deck full of secondhand information and hearsay. The quality of the information (or confidence in the information) fades a bit with each subsequent generation of copypasting. Sort of like lossy data encoding. At larger organizations, as you might imagine, there will be many "generations" of each presentation. This makes the lossiness issue even worse.
The culprit here is always time. It's exactly as the author of this article points out. People don't set out to make shitty decks, or to copy and paste information, or to tl;dr their way through analysis. But that's just what people have to do to survive. If you're spending (conservatively) 20-30 hours a week in meetings, and many of those meetings leave you with deliverables on tight deadlines, you're going to be eking out any and all efficiencies that come your way.
I've never done any copypasting, and if I'm being honest with myself, it's probably hurt me more than it's helped me.
In support of that, lifting right from the article,
Former CIA Director Michael Hayden once famously said, “If it is a fact, it ain’t intelligence.”
Man this could not be more true. You can get people to (sometimes) understand basic probability if they don't have a math background, but try something like bounded probability for explaining the likelihood of the severity of an event and they'll get lost.
The problem to me is that the same chilling effects we have seen develop in the journalistic community have also grown in the intel community, and when you have analysts who spend more time kissing ass than calling people out bullshit like they should, we are only going to end up in worse geopolitical situations.
NSA right now holds cards over CIA, keep that in mind as well.
Can you elaborate a bit or suggest some reading on that point?
The main reason I say though is because NSA has the upstream under their thumb. Pretty much anything the company does is going to be visible to the agency but not the other way around, which creates an information power disparity that is inevitable with the current organizational structures.
(Source: "Blowing my cover", the memoir of ex-CIA agent Lindsay Moran, who says that hearing about this was the reason she quit.)
Can you elaborate on this? Is this based on first-hand experience, media coverage, or some other source?
Some would balance this shoddy record against the assumption that "well we have to have some top-secret intelligence-sabotage-and-kidnapping service." That simply isn't true. Through most of USA history we didn't have anything like that at anything like the CIA's scale. National leaders do need to know about other nations, but that work could be done better by normal non-classified and accountable researchers, who at least in the beginning would not be wedged so solidly in the pockets of the military-industrial complex. The sort of intelligence that enables military action has been really poorly executed by the CIA, and we'd be better off without it.
It is important to remember that any "may", "likely to" and "could" statement can be replaced with a "might not" statement with precisely the same meaning, unless you quantify the probabilities (difficult and often so imprecise as to be pointless).
We see this all the time in tech journalism: "New technology might not lead to breakthrough in flying cars!" has exactly the same meaning as "New technology may lead to breakthrough in flying cars!" but for some reason we always see the latter and never the former.
I recommend reading any article with unqualified "may" statements with them flipped to "might not". It will rapidly become clear how much of this kind of journalism is hopeful nonsense.
Except the ending. That sounds like someone who has spent too much time around politicians and become infected.
It would have been interesting to see at least some acknowledgement of, and response to, the fact that the CIA currently has acquired itself a reputation for extracting false intelligence clumsily through relentless torture and brutalization. How, despite all the very smart and experienced people working for the organization, does that come to be the output? What goes wrong there?
EDIT: Why was this even posted? Useless garbage.
Human beings, who had spent centuries evolving away from slavery, were re-commoditized by the Industrial Era.
We are at the end of a five-thousand-year-plus historical process during which human society grew in scale while it abandoned the early indigenous wisdom councils and communal decision-making. Power was centralised in the hands of increasingly specialised 'elites' and 'experts' who not only failed to achieve all they promised but used secrecy and the control of information to deceive the public into allowing them to retain power over community resources that they ultimately looted.
We live in a constellation of complex systems. It is impossible for any single person or even any single organization or nation in isolation to understand complex systems.
It is our obligation to speak of what we know as we know it, not dissembling or deceiving. This could be considered the 'moral truth', and ultimately it is what can be validated by others so that a consensus can be arrived at and shared.
- Robert David Steele, 'The Open-source Everything Manifesto'
You sound anti-statist. Maybe you want stuff like this: http://mises.org/library/myth-national-defense-essays-theory...