hammerzeit

Reading these slides, I'm trying to parse what these slides do or do not say. I'd like to leave aside the speculation about what the NSA is probably doing.

First of all, XKeyscore seems to be primarily about the frontend query interface rather than the backend data storage, at least as far as I can tell. It looks like you can basically query their database by email address and get a set of records (email, chat, http logs) back. It looks like there are separate tools for viewing specific records as well. I assume they're joining records on some combination of email address, IP address, timestamp, etc -- not unlike a modern ad server.

A few practical thoughts: * It's worth noting what's not shown in these slides. Specifically, I don't see any ability to query the full text of emails. The more I see about this, the more I'm convinced the NSA is not collecting email body texts directly from corporate servers. Facebook messages I'm less sure of. * How are they collecting HTTP data? I assume intercepting at network hubs? * Given that it appears that individual records are HTTP requests, I'm shocked at how few requests are in the database. 41 billion seems an order of magnitude smaller than I'd expect. Could it be a record is something else? * Interesting to note the "Miranda number" and "Foreign Factor" fields that look like ways of saying "yes, I have permission to do this." Might explain why a sysadmin could bypass these things but your everyday NSA analyst could not.

Epistemology is not my strong suit so please excuse me here, but how can you reasonably advocate for your positions in a worldview that specifically privileges personal experience?

I tend to agree that we infer theory from anecdotes not data and even that that is a good thing. But don't we have a responsibility after developing a theory to evaluate it before making claims about it?

It's hard to comment on this post considering how many times we've seen the "HN-is-dying" meme, but for me this time it really resonates.

What has made HN interesting is the fact that it was boring to the vast majority of people on this planet. The topics were generally speaking on a very specific set of subjects that you largely had to be in the technology world (and of a certain technical disposition) to relate to.

The relative boringness of these topics led to comments by folks who found them particularly fascinating, setting a high community standard and discouraging soapboxing (comparatively speaking, of course -- I'm under no illusions that HN was ever the most pristine and idealistic place). For a long time I thought PG did a good job of enforcing that culture by shutting down off-topic conversations and keeping the focus fairly intense.

The problem with political posts is that they are often extremely relevant to hackers, especially these days. But everyone finds politics interesting, and everyone (myself included) believes their political beliefs are equally valid. Watching snarky posts and soapboxes get rewarded does send a message that this is how to get noticed. Long-term, I fear this fundamentally coarsens the level of discourse by making it easier to dismiss the beliefs of others and making it easier to participate in situations where I feel ignorant.

Unfortunately, I think there's only one way I know of to stop this -- which would be for PG and/or the moderators to take an active role in enforcing the "no politics" rule that's already in the guidelines. But I think they're perhaps legitimately concerned about censorship -- anathema to hackers -- and have kept a loose hand. Beyond that I'm not sure what they can do.

That sounds terrible, I'm sorry for your friend. Systemically that's broken and represents the potential for massive abuse.

That said, I assume you'd agree that anecdotes don't constitute data, and that we shouldn't develop strong beliefs on the backs of them. To assume the prevalence of something on the basis of loud anecdotes leads us to being easily manipulated -- in the absence of data I believe should keep our priors weak and seek better data.

I'm american as well.

When TV news and viral videos and newspapers make us feel like terrorism or kidnapping is more prevalent than it actually is, we call it sensationalism. Why is this any different? Is it because it confirms things you already believe to be true?

I looked at the list you provided, and tried to correlate it against the Wikipedia list. Many of the dates etc don't match up but I tried. A number of the cases where the suspect is listed as "unarmed" don't jive with the police reports, at the very least. Others involve things that are clearly assault even if there was no gun (like assaulting someone with a car).

That said, I do think in particular if you are a black male you are likely to be mistreated by police. That's not justifiable. But I still don't see real evidence that this is an epidemic or particularly pervasive.

I'm astonished by the total lack of conversation around the actual data here. Both the article itself and the discussion below are predicated on the assumption that this is a rampant problem in the USA, but not a single subthread on the actual incidence of this.

I'd expect better from Hacker News.

For what it's worth, Wikipedia has a complete list of people killed by US Police Officers and the police reports from them. In 2012, there were 587 people killed by police [1]. This is much, much larger than in Germany, even accounting for population difference.

I would argue that more relevant than pure population is differences in homicide rate. By the data you've given, approximately 1-2% of all homicides in Germany are by police officers (6-12 officer killings, 690 homicides in germany [2]). It's worth noting that in the USA, approximately 3% of all killings were by police officers (547/14748[2]) which seems in line with what my understanding would be -- it's worse in the USA but not ridiculously disproportionate.

[1] http://en.wikipedia.org/wiki/List_of_killings_by_law_enforce...

[2] http://en.wikipedia.org/wiki/List_of_countries_by_intentiona...

I read both of the original articles (also http://www.guardian.co.uk/world/2013/jun/27/nsa-data-mining-... ) -- there is still nothing in them that would cause this statement to be reliably true. You may surmise it, but speculation != fact.

Also, I think you must have a phenomenally short memory if you think Obama's policies on national security are stricter than Bush's were.

Don't you think if there was evidence that the government was still harvesting CCs,BCCs, etc en masse, that the article would have made this their central claim? The vast majority of the article is about how this stopped in 2011. One line implies they still collect "internet metadata" but not in bulk.

This headline is misleading. It implies the program is still ongoing, where the original article clearly states that the program was shut down 2 years ago.

Moreover, after a lot of cynical complaining about Obama not being meaningfully different than previous administrations, it's worth noting that Obama was the one to shut this down.

I'm not interested in reflexively defending the government or Obama but we still need to pay attention to the facts at hand.

In what world do PMs actually think the supposed thoughts in this article? Yes, it's important to document well, focus on why not how, and be a shit umbrella not funnel -- though these have been PM 101 everywhere I've worked.

But when PMs infringe on these issues (and yes, all PMs do) I find it's rarely a conscious decision as much an impulsive/instinctive thing from being torn in 100 directions at once.

Can you elaborate on a few of your claims here?

First, you seem to imply that a privately-owned public space is a bad thing. What alternatives do you envision? Wouldn't a space regulated and controlled by governmental entities or NGOs be just as liable to abuse?

Second, when you speak about "Facebook," what do you mean? Do you mean that the people who work there don't care about helping people connecting there? That Mark Zuckerberg personally doesn't care? Or that the corporate entity of Facebook has some sort of conscious will of its own separate from the will of its employees?

Third, you seem to imply there's an obvious reason why Facebook blocked TOR that furthers their business interests. Can you elaborate on what that is?

I do think there's something to the argument that any reasonable implementation of "Do Not Track" would just further entrench the companies who already have large consumer traffic (Google, Facebook, et al), by virtue of the substantial privileging of first-party cookies over third-party cookies at that point.

Now, you could argue, this is desirable -- after all, companies with large consumer brands have much more at risk in terms of doing unethical things with data (yes, I'm aware of the irony of saying that after the PRISM issue, but I think that only affirms my point). At the same time, does Google/FB/etc need any more help in dominating advertising than they already do?

I fear a HN with more glenn greenwald articles is a HN with more political flamewars. I don't think this is desirable, even if he does good work.

I certainly don't, and I think these are important and fascinating questions.

Do you have any evidence to suggest the definition of violent crime has changed at all? I'd be very curious if that were the case. In the absence of evidence, though, I'll continue going with what I understand to be the generally-accepted theory that crime rose dramatically throughout the 1970s and 1980s.

My fundamental question here comes back to this: If the number of reported crimes in the country doubled, what would you expect to happen to the prison population?

In the time period that the incarceration rate in the USA grew 4x, the number of violent crimes in the US grew nearly 7x [1]. This was a much more dangerous country then. The incarceration _per crime_ rate is only about 25% higher than it was in 1960 [2].

None of which to say there is not something deeply, deeply flawed in our country today and in the way we deal with imprisonment. Similarly there are real fundamental issues of race and class that make crime and enforcement problematic.

That said, we seem to look at institutions more than society when we look at social ills. But just like with healthcare and education, the two are inseparable -- we must consider the priors.

[1]: Data sourced from two sources, focusing on 1960-1994: http://www.disastercenter.com/crime/uscrime.htm and http://en.wikipedia.org/wiki/United_States_incarceration_rat...

[2] http://www.crimeandconsequences.com/crimblog/documents/pictu...

One thing I'd think about: If Google is lying or misrepresenting the truth here, then it would be phenomenally foolish to put out a denial under Larry's name as it just did.

Specifically, the personal integrity of the founders -- especially vis-a-vis these kinds of issues -- is one of the bedrocks of Google's culture. If a statement like this was proven out to be a deliberate misrepresentation (even if not an outright lie) it would cause IMO severe harm to Google morale.

As a user, I think I'm excited by this, although a lot will depend on the core usability and accuracy -- this is where this kind of technology often founders.

As an entrepreneur, this seems phenomenally relevant, given the relative penetration of GMail among early adopters. The choices that GMail makes can, within limits of what users will tolerate, create new behaviors for its users and eliminate old ones. This can create landgrab opportunities for startups capable of harnessing the new forms of attention and be destructive for startups addicted to the old forms.

I'm willing to guess that GMail is among the top 3 most used products by HN's readership. This is a substantial change in the experience of that product, as well as deployment of what seems to be some pretty nontrivial technology powering that. It's also the kind of feature that will impact any startup that uses email as a communication channel with its customers (to wit: All of them).

If that's not relevant to the HN community I'm not sure what is.

My initial instinct was to criticize this. I think that's some combination of feeling like hackathons are often a coarse method of exploitation and the feeling that this kind of boosterism may be overstated or unnecessary.

But the truth of the matter IMO is that this deserved to be recognized and celebrated. It's really exciting to see environments that expose people to the joy of making things and the realities of making things happen. This kind of culture was foreign at the major liberal arts universities I saw when I was an undergrad 5-10 years ago, and I think it's a positive change. Congrats!

Cool! As someone who was pretty critical of the FRC post from yesterday I want to make sure I spoke out to praise this. This kind of data, rather than an index, is much more interesting to me.

A few other jumbled thoughts:

* How are you tracking these? I assume it must be pretty messy collecting this data and keeping it up to date.

* Could you do something with Google Trends as well?

* Clearly some of these (whereberry/swish/etc) are highly correlated across channels, implying some sort of news event. Would be really interesting to see the ones that only gain in one without the others, and what could cause that. This kind of turns into Google Finance for startups.

* Agreed the LinkedIn data is intriguing, if only because it has a bias towards much more established companies -- maybe because it's focused on job-seekers?

* Lastly, one philosophical point: If you go by the mantra of "what gets measured gets managed," does this end up reinforcing the silicon valley hype cycle and encourage startup CEOs to get "hot"?

I'm a big fan of first round in general, and this is a cool idea and great data. But this index seems to be extremely sketchy, especially for B2B companies. The article says:

"This index is built on a weighted points system which considers the following factors: website traffic, social media following, social media engagement, employee count, page rank, inbound links, and SEOmoz domain trust score. Funding information is provided to give the reader context, but is not factored into a company’s ranking position."

First of all, many of these factors would seem to be correlated and self-reinforcing (is the ranking data published somewhere?) Also, the vast majority of these factors ultimately come back to publicity -- which is not necessarily remotely correlated with success outcomes. Companies like Invite Media, which certainly qualify as a substantial success, would rank low on this index.

Ultimately, this index seems to reflect less about which startups are actually sound businesses per se and more about which ones are generating excitement and hype. Ultimately I'd be more excited to see an index doing the former not the latter. Are the data sources for this linked to and/or explained anywhere in greater depth?

Eric Schmidt has a Ph.D. in Computer Science, wrote Lex [1], and just spent 10 years running one of the most important software companies in the world, while generally retaining the respect of the thousands of engineers in his company. His credentials are as close to unimpeachable as, say, PG's.

Tech is massive, multifarious, and constantly growing. It's easy to believe one's parochial corner is the center. Tor, it's safe to say, is not the center.

[1]: http://en.wikipedia.org/wiki/Lex_(software)

Well said -- as you imply, it's much scarier to contemplate we have the government we more or less want than to think it's being imposed on us by shadowy forces.

To add on a few additional points of evidence: 1) In the most recent set of national elections, there was not much correlation between money spent and probability of victory [1]. While second-order effects may still be problematic, I think it's worth reemphasizing.

2)Congressmen doing things that are against the will of their constituents leaves them massively open to primary challenges. See what's happening with Chambliss or other republicans in the recent budgetary crisis.

3)I do think there is a particularly pernicious form of corruption in Washington, stemming from the fact that lobbyists and other non-governmental entities often draft a large number of bills. This doesn't manifest as a specific national policy goal not being achieved, but rather causes subtle aims to be achieved in quiet fashions. This is not a problem of money, though -- it's simply a function of availability, and the convenience factor of who sees whom on a daily basis. Perhaps this could be fixed by empower congressmen and senators to carry larger staffs (staves?) than they do today?

[1] http://sunlightfoundation.com/blog/2012/11/09/how-much-did-m...

Articles like this are toxic for a community.

First, it's largely unsubstantiated, predicating most of its arguments on insinuations without much substance if any in the entire article. Without the usual navel-gazing about Journalism, it's still worth pointing out I wouldn't associate trustworthiness with any brand willing to publish an article as poorly-sourced as this.

Second, it ascribes to single-minded malice what is much more likely a vast and complex series of behaviors, some of which probably involves spammers but much of which does not. This kind of reductionism is rarely constructive IMO.

Most problematically, it lays the foundation for the delegitimization of dialogue in a community. This kind of paranoid accusation soon empowers folks to stifle dissenting voices as part of this outside group trying to bring down reddit. This kind of paranoia has already infected dialogue on the middle east in reddit, for example.

When they say video, they mean sites like YouTube etc -- spot TV is so negligible as to be pretty much a non-factor. As you can see from the 2012 report (http://www.iab.net/media/file/IAB_Internet_Advertising_Reven...) video is < 5% of overall spend and is not rolled up with display. The number you should be looking at is Banners + Rich Media, which is what most publishers use.

I think it's important to stress that this is totally incorrect -- the substantial majority of display advertising is still being paid on a CPM rather than CPC basis.

Look at this data, for example: http://www.adexchanger.com/online-advertising/iab-2011-onlin...

"Performance pricing" (CPC and CPA) was 65% of all online ad spend in 2011, but if you remove search from that, it goes down to 20%. CPM (per-impression) pricing is around 33% of all online ad spend.

Block ads if you want, but you should be aware you are depriving publishers of revenue if you do so.

A solid essay and a great set of criteria. When I was a PM at Google, there were a couple other critical traits I liked to look for, though they were not simple yes-no criteria.

* How does this person deal with vast uncertainty? Are they excited? Scared? Is their first reaction to impose structure or are they comfortable with lack of structure? If it's the latter, can they meaningful convey confidence with the lack of structure?

* Can this person teach? Can they figure out how to take an abstract idea and articulate it cleanly? When someone doesn't get it, at what point do they fall back to utilizing authority? Are they good at listening closely to what someone is saying to them and framing an argument in their terms?

First, I find it's unproductive to make psychological diagnoses of people you haven't met before. Having met actual pathological liars there is not remotely enough evidence that Lehrer is one, unless you're just using the word to mean "a liar I don't like," in which case you're cheapening the phrase.

Second, do you really want a world where errors -- even massive ones -- automatically disqualify you from being relevant again? Where there are no second acts in American life? I do think there are places where that makes sense, but they tend to be places where the supply is perceived to be abundant and commoditized. Ergo why we encourage failure amongst entrepreneurs.

Nobody automatically deserves forgiveness, but I do think Lehrer is a giften enough storyteller that we should not necessarily consign him to the dustbin.

Leaving aside one trolly user, most of the conversation here is either referencing data mentioned in the article or asking questions/having discussions about semi-related points.

I'd hardly call the argument over language speed a 'flame war' by any internet standard I've ever seen.

There are probably better examples than this one to pull the 'HN commenters suck' card. The conversation here is mostly civil and trying to learn things, one troll aside.

This headline is a little misleading, in that it suggests Abe went from being an engineer at Google to being an engineer at the Tribune. My memory is that Abe was in a technical specialist role at Google -- deeply involved in technical things, writing code sometimes -- but not in the formal engineering organization.

None of that harms his points, which I think are quite sound, and the actual article does not imply Abe was a full-time engineer at Google, but the HN title does seem to connote it.