Comment by username90 | Hacker News Reader

Comment by username90 | Hacker News Reader

Parent Full thread

> one participant was excluded because he was an extreme outlier in learning rate (>3 sd away from the mean).

Can anyone explain why did they do this? Was it just to prove their hypothesis? I bet this guy had really high numeracy scores as well.

Edit: They also botched some of the numbers:

> (Fig. 1A: mean learning rate = 1.25, range = 0.81–2.0, sd = 0.24)

The 2.0 point is not included in Fig. 1A, the highest learning rate is 15 lessons over 10 sessions or 1.5, so they included the outlier in that range but removed it in the plots.

Also I don't find it unreasonable that a person completed 20 lessons compared to the others 15, so I don't see why this was so extreme that he had to be removed. You could argue that the later lessons are harder so it was unreasonable, but then you just confessed that the learning rate metric is not linear so your regression models doesn't work since it assumes that going from .5 to .6 is equal to going from 1.5 to 1.6! Even though as we can see from the plot many of the best learners got stuck at lesson 16 for an entire session without making progress, while the slowest learners took 10 sessions to reach roughly the level these fast learners reached after the first. That is almost a 10x difference, but this plot shows it as just 2.5x.

Removing outliers is a common practice in data science. A quick search on "why removing outliers" returns a lot of discussion on the topic of when it should be done and when it should not. I encourage you to read some to get familiar with the arguments.

In this specific case, I suppose they could have added some motivation for their decision, but I don't think they did it in bad faith. They only had 36 samples, and a 3sd event occurs once in ~300 cases: this hints to the fact that that data point might have been erroneous, or that there could have been some other factor (e.g. earlier exposure to programming concepts - even disguised as something else).

If you are interested in the study, you can find the data and the script in the linked repository: https://github.com/UWCCDL/ComputerWhisperers The range for the learning rate seems wrong. If you check the data, there is no data point with 2.0: https://github.com/UWCCDL/ComputerWhisperers/blob/master/Com...

> They only had 36 samples, and a 3sd event occurs once in ~300 cases: this hints to the fact that that data point might have been erroneous

Or it hints that the distribution of learning rates is not gaussian. When there's an "n-sigma" event, it's usually much more likely that the model is wrong than that the event is that rare.

> Was it just to prove their hypothesis? Yes

Removing outliers is common, but extremely sketchy. The existence of outliers in the dataset at all is most likely because they are assuming a probability distribution that does not hold. If they are using the wrong distribution, none of their results mean anything.

> Can anyone explain why did they do this? Was it just to prove their hypothesis? I bet this guy had really high numeracy scores as well.

It's likely a predetermined exclusion criterion (although they should explicitly state that). An extreme outlier on the primary measure may suggest that the subject was not doing the task correctly (resulting in an extremely low score) or it may suggest that the subject misreported their lack of programming knowledge (resulting in an extremely high score).

Depending on what you're looking at, it can be standard procedure to exclude extreme outliers, but it should be announced as predetermined and be principled.

With statistics you want to generalize. Find the simplest model that catches/explains most variation. Leaving that one outlier in would result in a worse model.

Reply on news.ycombinator.com