Hacker News Reader

Comment by jfarmer | Hacker News Reader

This is probably worth reading: http://en.wikipedia.org/wiki/Selection_bias

Selection bias exists, but it doesn't explain away numbers like "700,000" and "94%". The fact is that Facebook pissed off a huge number of their users, for seemingly no good reason, as the previous design was quite popular and functional.

First, 700,000 isn't "huge" for Facebook. They gain about that many new users in a day.

Second, the number of respondees or the proportion responding a certain way are irrelevant if the sample is biased. The poll is designed in a way that practically guarantees bias: a self-selected sample, a hot-button issue where one side cares much more strongly than the other, and the viral hooks of Facebook to make sure it gets spread far and wide.

It's worthless if you're looking to draw any real conclusions about how Facebook users feel about the new homepage. It's not worthless if you're looking to exploit people's confirmation bias and get an article all over the web.

So, YMMV, I guess.

Reference for ~700,000 new users per day

http://www.insidefacebook.com/2008/12/16/facebook-now-growin...

Thank you for this link.

It also reports 13 million users log in once per day. That, in the context of Facebook, is what I'd call an "active user". These, overwhelmingly, are the people who are taking this poll. 700,000 therefore represents about 5% of the core userbase.

> 700,000 therefore represents about 5% of the core userbase.

Your math here is really flawed. Facebook's daily uniques are a magnitude smaller than their weekly and monthly uniques. Since this survey has been up since the redesign, you would have to compare 700k to the number of uniques over that period of time. I can assure you that number is much greater than 13M.

I'd bet you a dollar that at least 3/4 of the poll-takers are dailies. These are the users who really are taking this personally.

The whole point of this argument has been over statistical invalidity. Your guesses are pointless without data.

What? Your logic is as if all users knew about this poll, when I guess that the number of people checking for new apps is extremely tiny compared to the entire user base. I don’t even knew about this app, and it's more likely that 700k is very close to how many actually look for new apps on facebook during this period.

Another point: of Facebook's 175 million active users, 700,000 represents 0.4%.

Here, "active user" means "registered account". It's not a good indicator of who is actually using the site on a regular basis.

Generally FB takes "active" to mean "logged on within the last 30 days." That includes new users and returning users, but not users who haven't been on for the last month.

Please, if you're going to act as a proponent of responsible statistics, don't exaggerate like you did in your first line. (EDIT: wow, this is not an exaggeration. An adjacent link confirms it. Currently eating my words.)

True, 700,000 is not a staggering number for Facebook, but is a statistically signifiant portion of their most active users. I wouldn't put money on the figure of 94%, but I think it correctly indicates a strong trend.

I didn't exaggerate anything. Two years ago they were growing at 200-300k users per day. Dave & Ami were using those numbers in public talks. So, let's be generous and say instead that 700k users represents two or three days worth of growth rather than one.

It actually doesn't matter with respect to the more salient point, which is that talk of "statistical significance" is irrelevant because the calculations require an unbiased sample. As I outlined above, there are at least three strong reasons to suspect this poll is irreparably biased.

Self-selection alone is typically enough to condemn a poll. Look up the history of Gallup if you want to see the real-world impact of self-selection in polling.

selection bias is compounded by the fact the costs of voting. i like the redesign, and i was going to vote YES, but it's not a Facebook poll, it's an application. I'd have to sign up to use the app, allowing it access to my user data, just to vote. No thanks. I bet most people who like the redesign don't want to jump through all the extra hoops to vote yes.

Yeah, people who are happy with the new layout (or don't care) are not likely to install an app just to respond to the poll.

I took a few days before adding the app to vote yes, and I just did that because I was pissed off at the people voting no.

For the record: perhaps 10 of my friends have voted no. This is not a significant percentage. Of those 10, probably about 8 of them did it as an impulse vote and 2 actually got annoyed at the layout.

the number of respondees or the proportion responding a certain way are irrelevant if the sample is biased.

Correct. Ordinarily, the ABSOLUTE size of a sample is more crucial than what percentage of the whole a sample makes up, for determining whether the sample is adequately large, but that is only if the sample is unbiased. If that precondition is not met, don't talk about how large the sample is, talk about how badly self-selected "samples" can misrepresent the characteristics of a population. Thanks for reminding HN readers about this important issue.

The one thing you're forgetting is that this happens _every_ time there's a new website redesign for Facebook. Its actually very typical for this to happen in general. People despise change will complain about a new design even if the design is much better. In a couple of weeks no one will even notice or think much of it.

Everytime single time that Facebook has changed the design almost everyone hates it and everytime almost everyone grows to like it.

>the previous design was quite popular and functional.

But there was just the same outrage over that design for a good month. I probably got 10 group invites to groups like "new facebook sucks!" and "don't visit facebook on (some date) to show them how much we hate new facebook."

I'm not sure if anyone took a poll, even a biased one, but I haven't noticed among my friends any more outrage over this redesign than the last one.

Everyone hated the news feed, yet now it's hard to think of wtf Facebook would be if it didn't have the feed. Everyone hated the first major redesign too, but now I'm sure everyone's used to it. They should do a test, wherein they switch the layout back to a previous layout then have people vote. I have a feeling this would show that the vocal majority would dislike ANY major change. I'm pretty sure Facebook's team has a lot better designers than any of the 700,000 voters.

Reply on news.ycombinator.com