Hacker News Reader

Comment by Jiro | Hacker News Reader

The answer has to be A, under reasonable assumptions. If you assume that the university wants to find out the opinions of all students, and if you assume that students who didn't respond still have opinions, then the problem is that the students who didn't respond may have different opinions from the students who did, skewing the results. Only choice A tries to solve this problem.

People here are drastically overthinking it. Yes, it's true that the problem didn't state those assumptions, but you are supposed to make them, not to say "the university didn't give us its criteria for 'best plan' so all the answers are equally good". Given the way that problems like this work, it's obviously meant to test knowledge of the specific idea that the people who respond may be nonrepresentative of all the opinions. Only response A shows knowledge of this idea.

View on HN

"Has to be" is a stretch. By nagging the non-respondents you are introducing a bias that the first group of respondents were not exposed to. If you annoy a respondent, it's very reasonable to assume that can influence their responses (probably negatively).

Option D seems objectively better to me over Option A based on this bias.

Option A is strictly superior to option D, as you are still have the data for original 90 responses. If second round responses are similar, you have increased confidence that you are not running into responder bias. If it's different, you can publish both and note uncertainties and need for further study.

I hear your argument, but that also introduces a slippery slope of cooking the data to support your original hypothesis. Now that you have the original data set and the expanded data set, one is likely to do that better than the other.

If the research industry were as pure as science itself of course that would never happen...

I actually vote against A. My work was always sending out 'completely anonymous' surveys that I generally ignored.

Resending to tell me I hadn't done it yet was indication enough to me that it wasn't completely anonymous at all.

While the question doesn't mention anonymity specifically, I'm sure it remains a point in some percentages mind.

But I could make a table Request(email, url, submitted) and a table Response(data)

And you couldn't connect the response to the user. Thats the important part right? not the ability to resend a request

I'd argue recording anything at all, including whether I did it, breaks the 'completely anonymous' contract.

Then we're left with...mostly anonymous. But at that point it's a black box. It could be done how you describe. Or it could be attaching my name to the survey and emailing the CEO directly. I'd never know the difference.

I mean sure. If you think someone is lying to you then it doesn't matter.

But I disagree with "getting a reminder means its not anonymous" its completely orthogonal.

Are you assuming the request to respond is broadcast to all recipients regardless of response status?

No implementation is something like email is primary key, url is per user.

going to page and submitting will do two things. the table Request will be updated and marked submitted. the table Response will be populated with only the data.

If you can derive the email at submission time then its not anonymous.

It seems like the disagreement is between what makes data anonymous.

If I write a letter and don't sign it. It's anonymous. Someone could use a corpus of my text and infer I wrote it. That doesn't mean I didn't write anonymously.

I could make 2 updates to 2 tables and then end result would be that having both tables wouldn't let you correlated the data with submitters.

Yes if you control every aspect of the process you can lie to people. Thats not the point. If you think someone is lying to you to harm you why would you interact with them?

The point is there's ways to achieve this even assuming bad actors and doing something that's obviously flawed doesn't seem helpful.

> obviously

The question never says the survey is or claims to be. completely anonymous.

I work for a company that actually runs surveys, and believe me: we don't know who the respondents are. They get a link with a randomly generated id, and that's it. For us, even their email address is hidden, because respondents register to panel at another company, which deletes this information three months after closing the survey, which usually runs for a few days. I can't vouch for the other company, of course, but I do believe them. Sometimes it would be really nice to know who these people are, because not only is non-response a problem, bad responses are an even greater problem, so we'd like to know who to exclude in advance.

So much for unbiased representation.

Two fallacies:

1. The non-responders are not responding for similar reasons.

2. Those reasons are somehow related to their opinions.

Both are clearly false. You cannot assume anything about their opinions from the fact that they didn't respond. All you have is their (non)action - which could have been caused by being ill, busy, distracted by personal issues, absent, drunk, or for any other number of reasons.

None of the answers are correct. If you wanted representative data you would have to:

1. Confirm with a different sample but also...

2. Use multiple different channels and methods (phone, in-person survey on campus, maybe an email survey with some kind of benefit to encourage participation, etc) to try to eliminate the non-responders who have personally valid but statistically irrelevant reasons (away, ill, etc).

As the blog post says non-responders do tend to have different opinions on average than responders. Typically they do try to demographically determine who those non-responders are so they can adjust and increase the weight of that group proportionally to their limited response. This is how most polling works in the real world and it does provide better accuracy.

Suppose the non-responders are identical to the responders. Then nothing is lost by trying to get them to participate again. But if they are not, replacing them will lose information.

> ill, busy, distracted by personal issues, absent, drunk

But being ill correlates (lightly) with being older, and being drunk heavily correlates with opinions. Being absent might correlate with wealth (and not even linearly). Personal issues probably correlate with education. Add them together, and you've got a fairly large part of the population missing from your sample.

>The answer has to be A

We have this problem at work. You try to contact customers and they never respond. Sending repeated queries has no effect.

The real answer is: The question is poor and ambiguous, therefore the question is wrong. It reflects poorly on the academic institution asking it. I would be in search of a new school for my child if I saw many test questions like this. They didn't even enumerate the answers correctly.

It is asking for the best plan of action for the University without clearly specing out the goal and all its parameters. The goal may be "Determine if at least half of our first year students had a positive experience." and that can possibly be solved with 90 responses if at least 60 had positive experiences.

It's just like when your boss asks for a time estimate on a task he's barely defined. It depends on the spec he's asking from you. Without clear specification, the question isn't answerable.

> It reflects poorly on the academic institution asking it.

Only in so far as it is posed with a multiple choice response and no opportunity to make a comment.

The question itself is about a commonplace occurrence in statistical sampling.

>The question itself is about a commonplace occurrence in statistical sampling.

It certainly is, but once again, the question does not specify enough information to choose an answer. Maybe A is correct, if you have a backup/secondary email to try. Maybe, like so many of us who work with email notifications, they can see the email wasn't opened. Or it bounced. Or it was opened and flagged as spam.

Is that all the information you have? You emailed 120 of them and received 90 clickthroughs? Well sir, that is an astounding clickthrough rate! Perhaps that exceeds every expectation of responses and is itself actionable. Perhaps the best plan of action for the University is to discover their magic clickthrough sauce and monetize it.

What is the best plan of action? Impossible to answer when they haven't specified a goal.

You may have missed the part where I pointed out that it's a fine question .. that requires an opportunity for comment.

It's not a multi choice question (a form I dislike) but a question that requires a reasoned response.

>it's a fine question

>It's not a multi choice question

But it IS a multiple choice question, and it is ambiguous, which is why it's not a fine question. It's a sloppy, poorly constructed question, which isn't even enumerated correctly, allowing the instructor to select the "correct" answer based on unspecified reasoning, potentially discriminating against the person answering the question (which I suspect is the intention).

> which isn't even enumerated correctly

You keep saying this, but the screenshot shows that the student has used an elimination tool (the X on the left) to eliminate one of the possible answers (C).

> allowing the instructor to select the "correct" answer based on unspecified reasoning, potentially discriminating against the person answering the question (which I suspect is the intention).

The question looks like it's from albert.io, so the answer is predetermined (and generally provided by the vendor).

I have no reason to believe that students that ignored the survey the first time would respond to a second survey. There will always be non-respondants, it's just part of the domain. I'd vote for sending out 120 more surveys simply because 120 is a very small sample size to begin with.

There’s always a risk that if you increase sample size without addressing selection bias, you’ll just end up with more confidence in an incorrect conclusion. (“Dewey defeats Truman!”)

Right, you also don't know what was covered in the class. But do the best you reasonably can to collect a random sample and then go with it is almost certainly the best answer in this case. (Although you can make an argument for D.)

Option A is absolutely the worst choice. The non-responders have already selected themselves into a non-random set, and making a request to them alters the entire experiment.

Surely the responders are also self selected into a non random set too?

The set you wanted, so IN the experimental design. Or did the experiment include also begging people to fill out a questionnaire they had initially trashed? Many people appear to think so.

The people who didn't represented when nudged can answer incorrectly because they are just trying to speed through it. It should be treated as a separate population and not merged into the rest.

This is how we do it in reality. Missing answers are very important!

No, the question says the original 120 were "sampling". There is no "all".

> Only choice A tries to solve this problem.

No, it doesn't. It doesn't get any information from non-responders; it can't, because they don't respond! It only gets information from some people who respond on the second try instead of the first.

There is nothing in the question which suggests some of them (or all) wouldn't respond to a second, third, or nth prompt. In many surveys, non-respondents receive multiple outreach attempts.

> There is nothing in the question which suggests some of them (or all) wouldn't respond to a second, third, or nth prompt.

Of course not. But making second, third, or nth requests is not "sampling from non-responders". It's just creating more and more different groups of responders. Which doesn't improve anything statistically that I can see; in fact it might make things worse, because you are introducing more and more potential confounders.

When this is a concern, I do two analyses -- one with and one without the potentially confounding group. It is still better to obtain the data, since otherwise we knowingly have a biased sample. (Somewhat similar to only surveying people who "seem approachable.")

> When this is a concern, I do two analyses -- one with and one without the potentially confounding group.

If this analysis tells you there is no significant difference between the subgroups, that's nice, because you can then just lump them into one and not worry about it any more. (This also tells you you didn't actually need to send out the second email, but what's done is done.)

But if this analysis tells you there is a significant difference between the subgroups, you don't know if it's something that was there already or something that you created by sending out a second email to people who had not responded to the first. So you're not actually better off than if you hadn't sent the second email; you're worse off, because you now have a confounding variable present and you don't know what to do about it.

> otherwise we knowingly have a biased sample

You have a biased sample regardless of whether you send out a second email or not.

It seems you could possibly apply this same logic to exclude any subgroup.

For example, you must sample the mall-goer population. Some people "seem approachable"/etc. ("respond to your first email"), and you approach them. You are worried that others will be annoyed if you approach. If this assumption is true (which it may not be), their responses could be biased and confounding.

Should you still survey them? I believe the stats textbooks would say yes, since it will result in a more representative sample.

I assume it would be a detriment to the survey if you exclude some groups merely because they aren't compatible with your survey-taking techniques. All surveys have some inherent measurement error, and there are some techniques to detect biases/inconsistencies without excluding entire groups.

> It seems you could possibly apply this same logic to exclude any subgroup.

No, just any subgroup that, by construction, is not reachable with whatever technique you are using. The key point is that there will always be such subgroups. See further comments below.

> Should you still survey them?

If you mean, should you still send out surveys the first time even though you don't know how people will respond? If you're going to do a survey at all, you have no choice, so this question is pointless.

If you mean, should you continue to pester people who didn't respond the first time, on the grounds that "it will result in a more representative sample", I would question whether that is really the case. See further comments below.

> I assume it would be a detriment to the survey if you exclude some groups merely because they aren't compatible with your survey-taking techniques.

It's a "detriment" if you fail to realize that there is no way to avoid it. Taking a survey inherently means you get responses from people who respond to surveys and not from people who don't. Going back and asking a second time for a response from people who didn't respond the first time inherently means you get responses from people who will respond the second time and not from people who won't. And so on. There is no way to avoid the fact that any survey-taking technique will inherently exclude some proportion of the population.

Once you realize this, you realize that the idea of "getting a more representive sample" is based on a false premise, that you can somehow "include" everyone if you just go about it the right way. You can't.

"Getting a more representative sample" is premised on getting as representative a sample as possible, not necessarily perfectly.

Using language such as "pester" is making assumptions about the group. In fact, many likely were busy or have just forgotten to reply. As I mentioned above, there are techniques for assessing biased/inconsistent responses.

> "Getting a more representative sample" is premised on getting as representative a sample as possible

But representative of what? You can't get a sample that is "representative" of non-responders--because they don't respond.

If you ask once and include whoever responds in your sample, your sample is representative (you hope) of people who respond.

If you ask non-responders again, your sample now includes people who responded when asked once, and people who responded when asked twice. But the "asked twice" part introduces an extra variable: did the fact that you asked them twice change something that you would rather had not been changed? Given that, it's not clear that the second sample is any more "representative" of anything useful than the first.

Representative of the population under study.

In practice, "extra variables" are naturally present in every study. For example, surveying mall-goers in the morning vs evening. Or surveying people at one end of the mall vs the other. These variables are likely far more confounding than "must send a followup email".

People asked in the evening might be more annoyed, but you shouldn't assume this a priori and decide to skip surveying them. Just as "busy people" in the studied population might forget to initially respond to an email and need a reminder.

Just as a personal anecdote, I've dealt with this issue quite a lot. In my surveys, the people who respond initially are almost always eager to give glowing reviews. If we didn't send follow-ups, we would have extremely positively biased results. Maybe it is true that sending reminders makes respondents slightly more negative than they normally would be. However, we'd completely exclude "unsatisfied people" otherwise and the survey results would be worthless.

> "extra variables" are naturally present in every study

That's true, but it doesn't change the fact that if you ask people a second time to respond, that's an extra variable you created, not one that was naturally there already. It's not a good idea to create extra variables in the course of doing the study.

It's true that the people who respond to the survey without any further prompting are not necessarily going to be representative of the entire population. But that doesn't mean you can "fix" that by further prompting. What it actually means is that surveys are a tool with only limited usefulness. That's just an inherent, inconvenient truth about surveys.

In terms of honest research, discovering uncertainties is as important as discovering trends, you just duly present these in your conclusions. If you are motivated to reach certain conclusions from the outset, you are looking for politics or marketing class, not statistics class.

Choice A will likely get some answers from students who didn't respond the first time, increasing the representativeness of your sample. It's only futile if you conflate as "non-responders" the people who didn't respond on the first opportunity with the smaller number who wouldn't respond even given a second opportunity.

> Choice A will likely get some answers from students who didn't respond the first time, increasing the representativeness of your sample.

Possibly. But it also introduces a potential confounder.

> It's only futile if you conflate as "non-responders" the people who didn't respond on the first opportunity with the smaller number who wouldn't respond even given a second opportunity.

No, the non-responders are the ones who never respond, period. Which, as I said, means you can't sample them to see what they're like.

What you're doing, instead, is creating two different subsamples of responders: those who responded on the first try, and those who responded on the second try. Which might make things worse, since now you have an additional variable that you created by sending a second email to those who didn't respond the first time.

Option A is absolutely the worst choice. The non-responders have already selected themselves into a non-random set, and making a request to them alters the entire experiment.

> The non-responders have already selected themselves into a non-random set, and making a request to them alters the entire experiment.

This is a better statement than any of mine of the point I've been trying to make.

Reply on news.ycombinator.com