Hacker News Reader

Comment by tomdell | Hacker News Reader

That is not the argument.

Tests are written by people who talk a certain way, in a way that is easier to process for people who talk the same way. Different communities use different language, and this is reflected in how quickly different students process the problems on standardized tests. There's also the issue that standardized test scores are higher for students who have the time and encouragement to study for them - things which are scarce in poorer families where the student might have to work or help take care of younger siblings. Years of institutionalized racism (loans not given, redlining, denial of employment opportunities) have economically crippled some ethnic communities.

View on HN

The diversity thing is mostly a red-herring. Admissions offers are abandoning standardized tests because of Goodhart's Law.

> for students who have the time and encouragement to study for them

It's not just time and encouragement. SAT/ACT prep is serious industry. $100+/hr tutoring sessions. Owners of good SAT prep businesses can make six or seven figures in a few months.

And good prep courses really do work. Tests, like coding interviews, are a game you can hack with enough time and effort.

The signal is totally washed out; with enough money, you can learn how to take the test well enough.

The research suggests SAT test prep has an effect of about one seventh of a standard deviation – about 50 points out of 2400[1]. That's significant but not huge. Most of the perception of large gains comes from the fact that people do better as they get older with or without coaching.

(None of the research is great, but better than nothing.)

[1]: https://www.jefftk.com/p/sat-coaching-what-effect-size

The College Board released new results in 2017 showing a more than 50 point increase out of 1600 versus students who didn't practice.

This was with just 20 hours of practice. Many high achieving students practice for 5+ hours per week.

South Korean cram school style 2-3 hours per day is in a completely different league than a standard SAT prep course, but much of the research that shows no result counts them the same.

This phenomenon is known as the practice effect. It has a ceiling in terms of IQ tests, just like it does in terms of physical tests. IQ tests are designed to be as close to ungameable as possible.

This is like pointing out that I can make large gains in my 100m sprint time by practicing for half an hour a day for a month and predicting that I can make the Olympic team if I put in a thousand hours practice. Novice gains are real and significant but they peter out pretty quickly.

IQ tests are gameable to a degree. They are accurate across large populations largely because most people don't practice, not because they are actually ungameable.

SATs are different in 2 respects.

1. There is a large incentive to practice.

2. It isn't an IQ test, it only correlates fairly strongly with scores on IQ tests.

SATs are highly knowledge based--most of the math questions can be answered by memorizing algorithms, and the verbal section depends heavily on vocabulary.

>This is like pointing out that I can make large gains in my 100m sprint time by practicing for half an hour a day for a month and predicting that I can make the Olympic team if I put in a thousand hours practice. Novice gains are real and significant but they peter out pretty quickly.

It's unlikely you could become an Olympic sprinter, but with enough practice you could probably move into the top 1% of everyone in your age group because the vast majority aren't going to spend that much time practicing.

The SAT equivalent (for the 2018 test) would be somewhere in the 1500-1550 range.

The SAT is highly g loaded and it correlates with tests that are designed and marketed as IQ tests at levels sufficient that saying it isn’t one is quibbling. The lowest estimate I found for IQ SAT correlation was an r^2 of 0.76.

SATs are more knowledge based than a test designed to be culture fair, like Raven’s Progressive Matrices but they’re not meant to be culture fair either, they’re for testing educated English speaking Americans. The test population are in high school but there’s nothing on the test that you wouldn’t have known by the end of middle school.

As far as algorithms and vocabulary go, most people have limited ability to follow complex algorithms because that requires a lot of working memory, which correlates with... Intelligence!

Vocabulary correlates with IQ about as strongly as SAT score, yet vocabulary can clearly be hugley improved with practice.

Even if correlation with IQ somehow proved score durability, a correlation of 0.76 or even .8 is low enough to allow for huge swings away from IQ in a very large percentage of the population.

A big chunk of students practicing and increasing their scores 200-300 points could easily be hiding in that 0.76 correlation.

>As far as algorithms and vocabulary go, most people have limited ability to follow complex algorithms because that requires a lot of working memory, which correlates with... Intelligence!

Most algorithms at that level are designed to be written out step by step explicitly to reduce working memory requirements.

Part of SAT prep is learning algorithms for solving word problems that reduce working memory load by memorizing patterns in the questions and writing down the components in a standardized form.

You said

>As far as algorithms and vocabulary go

And then never mentioned vocabulary.

It has nothing to do with working memory and most definitely can be improved dramatically with practice.

One more thing. The SAT was rewritten in 2016 to be far more content based, so the correlation with IQ will likely fall even further.

And good prep courses really do work. Tests, like coding interviews, are a game you can hack with enough time and effort.

Years ago I took the GMAT and scored very highly, 98th percentile, and GMAT-taking is definitely a learnable skill in and of itself. Then again, maybe the test is really a proxy for “willing to do prep work upfront”. Having said that obviously it filters for those who have the opportunity to do so, but I am willing to give the benefit of the doubt and say the latter is not it’s deliberate intent.

There are SAT prep books in the thrift store for a dollar. Nobody is shut out because of poverty.

Why are people jumping through hoops trying to imagine possible reasons for differences in academic performance when the best predictor is IQ and that can be measured from just a few years of age - before they would have to spend time studying or taking care of younger siblings. We already know that east Asian children have higher IQs on average than whites. That explains their better test scores at school. We also know that blacks have lower IQs than whites which also explains their lower test scores. So if you're looking for a cause of the different outcomes, you should look for the cause of different IQs. Since that starts young, maybe black parents don't hug their kids as much as Asian parents (that affects IQ) or maybe it's genetic (that affects IQ). The popular Raven's matrices IQ test has no words, just abstract images so there's no language bias there. Researchers have studied this to death over and over again for 100 years and they keep successfully reproducing the same results. They try everything to eliminate bias, such as having the test administrators be the same race as the subjects and controlling for socioeconomic status and going to different countries and controlling for the economic status of the country and so on. It's a solved problem down to the IQ level. What's not solved is how to improve the IQ of a person after they're born. You can improve it before birth by choosing parents with high IQs.

> Tests are written by people who talk a certain way, in a way that is easier to process for people who talk the same way.

So have your test writers distributed in a way that matches the general population. Don't just blindly try and force the population of VLSI design engineers or whatever to mimic the city's demographics.

Reply on news.ycombinator.com