Should the U.S. Make Standardized Tests Harder?
theatlantic.com
theatlantic.com
The equivalent where I am from is written essays and/or math, physics, chemistry exercises (ie. calculation) where you have to have all intermediate steps visible (which means you have an chance of scoring a few points even if the final answer is incorrect).
You take four to eight exams during a period of two weeks (opportunity to participate is twice a year), six hours per exam.
In language exams, there's a listening comprehension part (in addition to the six hours of written exam) where you listen to a recording and answer multiple choice or with a few sentences.
The method used here is not entirely problem-free either but it's a whole lot more practical than having a long string of multiple choice questions where you have a 25% chance by guessing and you get to know all the possible answers. This is not really representative of what you need in life or what you'll encounter in later studies in College/University.
The SATs are criticised for this within the US as well, but in truth I think they form better preparation for college. No international exam predicts university performance, but (as I recall) the SAT does no worse than any other.
In fact the US does a pretty good job of undergraduate studies. If 15 of the top 20 universities are in the US then something is working.
I will list a few reasons why I prefer the SAT system -
Foreign exams: "you have an chance of scoring a few points even if the final answer is incorrect". In this respect, the foreign exams are giving you credit for being wrong but well-intentioned. Is that a desirable quality? That you know what to do, but you can't do it in practice? IMO giving credit for partial answers to basic knowledge questions is a highly dubious practice.
Whereas, strategies for answering the multiple choice questions effectively require extremely quick and subtle logic. A lot of foreign students don't really understand why these multiple choice exams are difficult - until they realise they are failing. You have to be knowledgeable, and also logical about your knowledge, under significant time pressure.
The statistics of what happens to guessers in multiple-choice questions is very easy to examine (and they won't pass). The statistics of what happens to bullshitters in essays is not very easy to examine (they might pass with minimal knowledge, if they can fool the examiner successfully or hit upon a lucky essay).
The fact that the test is standardised and asks questions from a basic and fixed set of knowledge means that universities know exactly what set of things the arriving undergraduates will know, and can target the first year courses appropriately to first year students. This is something that other systems fail by a huge margin. I mean, it just doesn't happen correctly elsewhere - a majority of first year students are left struggling for a significantly wasteful period of time, because the universities have not understood what the core intersection of knowledge the intake have.
The SAT is simple enough (in knowledge terms) that US schools are free to teach any extension courses in any direction that they like, allowing much more creativity in teaching and learning. Most schools internationally have become exam factories focussed only on the final exams, and the retaking of those final exams.
Having taught both systems, I prefer the SAT system and I think it does a far better educational job.
None would give points for something that is wrong. They would, however, give points for using the correct technique in getting to the answer (but maybe something was missed, or miscalculated, etc). This is not being given "partial credit for being wrong", it's being given credit for having the technique to get to the answers you seek. In an age where data retention is a much less desirable skill than adaptation and the ability to find answers (what with Google existing), I think it's an extremely important point.
I think you are seeing the SATs in a good light because they succeed at doing what they are set to do: have students retain a bunch of information for a while. What they most definitely don't do is teach students how to think outside the box, think for themselves, approach problems they haven't encountered before and approach problems nobody has encountered before.
OK, but how can this be achieved by examination? This is really not what exams are about and it's a mistake to try and fit those goals into an exam space. Exams are considered unfair if they have an unlimited scope.
When I said the SAT gives more flexibility to teach off-syllabus topics compared to other international equivalents, this offers you a space where you can target those blue sky goals.
I don't criticise your goals, but exams are never where those goals are reached. So, I suggest we should minimise the concentration on exams. This is something the SAT system does (relatively) successfully - also because it is completed long before the end of high school.
You say exams are not about that and it may be so, but I'm not convinced it's not possible to make them about that. A giant list of multiple choice questions won't achieve it though, not any better than the briggs-myers anyway.
This is definitely a desirable quality in math/physics/chemistry exams. If you make a very long calculation (some of the exam questions require a full sheet of paper to calculate) almost perfectly but make a mistake on the second-to-last row, you should get your 4 out of 6 points or so.
Good point about language barriers, though. There are lots of smart people who have to participate in a test that is not their native language, in the US and elsewhere. History exams should be about knowledge and understanding of history - if your language skills aren't that great, you'll inevitably lose a few points because of the poor quality essay you can produce.
> The fact that the test is standardised and asks questions from a basic and fixed set of knowledge means that universities know exactly what set of things the arriving undergraduates will know,
Our tests are standardized too (and same questions nationwide, changing every year). They're just not multiple choice questions.
You make a good point about flexibility too. Our education in the few years preceding the exam were a bit too focused on the finals. I guess in that respect the SAT has a small advantage if they allow a more flexible curriculum in the preceeding years.
I question this. (I have worked as a teacher and as a practitioner)
First of all, should we ask such long calculations in an exam situation without access to a computer? That's not real anyway.
Second of all, if we are asking the long calculation with the expectation that students won't be able to complete it, why are we asking that question? Why don't we ask shorter questions that match the expectations we have?
If we are asking the long calculation because we expect students should complete it, why should they get partial credit for failing to meet that expectation? That's what you're testing them on - and they haven't truthfully shown they were better than someone who wrote nothing. They've failed at completing the task, and/or at checking their work.
It really shows they were better (or better trained) at gaming the exam - knowing exactly what to write to satisfy the examiner and gain partial credit. This training is very well paid, by the way.
>Our tests are standardized too (and same questions nationwide, changing every year). They're just not multiple choice questions.
Even if the questions are consistent nationwide, if the pass mark is low enough (or with partial credits), students can arrive at university with wildly different knowledge, having passed via different sections of the course, if not via luck. The university will naturally assume that the student knows all of those sections well, which many students will not. This creates an obvious problem of services being inadequate for first year undergraduates. Where I live this is readily apparent, but few people have noticed and nobody is dealing with it.
What if you make two mistakes that cancel each other out? You get the correct answer but should you get full points? In my opinion: no.
> First of all, should we ask such long calculations in an exam situation without access to a computer? That's not real anyway.
First of all, students have pretty fancy calculators these days. And one sheet full of equations is not that much for a practical, applied problem. Testing only mundane math skills (simplify, differentiate, solve, etc) doesn't test the ability to apply the skills to any real world problems. There are short problems too, it's not all long problems.
If it was only about the correct answer, the grading might as well be 0 to 1 points. Now there are questions worth 1, 2, or 6 points. The first being "simplify this equation" and the last being an applied problem.
And learning how to do math without the aid of a computer is a very valuable skill. In practice you have the aid of computers, books and collections of formulae but in my experience, there's no substitute for pen and paper practice. Learning to solve math problems is a skill, not a piece of knowledge.
If you don't know some things by heart, and know how to attack a problem, a computer will not help you.
> Second of all, If we are asking the long calculation with the expectation that students won't be able to complete it, why are we asking that question? Why don't we ask shorter questions that match the expectations we have?
The expectation, of course, is that a student will be able to complete the whole thing. But the longer the calculation, the higher the chance of making a simple stupid mistake. The reviewer will evaluate whether or not the student showed that he understands the problem and the solution. If there's obviously no idea what is going on, it's going to be zero points. Even one point requires a partial solution.
> It really shows they were better (or better trained) at gaming the exam - knowing exactly what to write to satisfy the examiner and gain partial credit. This training is very well paid, by the way.
No, that's not the case at all. Getting partial credit is completely worthless, you need to have at least 50% of the score to pass. Trying to "game" the exercises won't take you far.
For a point of reference, when I did my finals more than a decade ago, you needed 57/60 points (ie 10 problems) to get the highest grade. That means making at most three trivial mistakes during 6 hours.
We are asking that question, because we want to verify that the student understands the process of solving the problem. Therefore final answer is of lesser importance - what matters is proper procedure that is used to arrive to the answer.
And no, shorter questions and shorter answers are not always a way to go, because some problems do require lengthy procedure to be solved.
And of course you're right that in real life such calculations would be done by a computer. But that is because in real life the goal is to solve the problem, while at school the goal is to teach the student and verify that knowledge (so that in the future he will be able to create a computer able to solve such problem, metaphorically speaking).
> If we are asking the long calculation because we expect students should complete it, why should they get partial credit for failing to meet that expectation? That's what you're testing them on - and they haven't truthfully shown they were better than someone who wrote nothing. They've failed at completing the task, and/or at checking their work.
Again it comes down to the expectation - student should be expected to understand the problem and the solution, not to write a single number as an answer. Therefore partial solution (i.e. a procedure where some steps are invalid, but some are still valid) is better than no solution.
Just an example. A 'complicated' geometry problem. Our expectations is that the student understands the text, draws a draft of the shapes. She then recalls the appropriate two or three equitations. Formulates a plan to solve them (first this eq, which gives that info, with which I can ...), then she executes that plan and writes down the result.
Student A does a neat drawing, marks the important measurements, uses those same markings to note the equitations down and sets out to solve them. During a substitution she mistakes one of her 3-s to a 2, therefore the result is completely wrong. She completed 3 of our expectations and failed the last one.
Student B draws a happy seal and takes a nap. 0 out of 4 completed goals.
I would argue that the first students deserves more credit, and those are full credits not partials.
However, unless you are specifically testing the student's ability to string the different aspects together precisely, I don't believe that to be an optimal exam question.
Here, for example, you are seem to be allowing the student a 50% passing grade (2/4 expectations met) if they show they understand the text by drawing it out, and remember the formulas from the syllabus sufficiently to write them down without starting to apply them.
That maybe isn't your intent, and maybe your ideal mark scheme would try to avoid over-crediting something like that.
But even if that's not what you intended, that has become the reality: setting expectations for partial credit has led to increasingly simplistic expectations of students and this in turn to rampant grade inflation. Grade inflation has been strongly encouraged by institutions' statistical collation, but has been mostly avoided by the SAT.
I'm not sure that I want students to have access to good grades even if they consistently give wrong final answers, transposing digits and so on. It shows a lack of precision and a lack of full understanding of the question (not able to do effective sanity checking). Precision is a skill that mathematics can help people to learn, in my opinion.
The SAT being multiple choice means you have a fair opportunity to correct this type of mistake - a typographical error typically won't fit with any of the multiple choice answers and you'll have the chance to correct it (costing you only the correction time - and teaching you that precision is of some importance). The wrong multiple choice answers tend to fit deliberately with common conceptual errors. This actually illustrates a useful element of the SAT exam design.
However, calculations usually take months to set up and implement, and even longer to test, and each calculation needs to be checked and re-checked several times. There is pretty much no time I get the right answer the first time, and I am the go-to person for what I do.
It's certainly not perfect, but it's not the worst thing in the world. It does favor higher socioeconomic classes, though, for a variety of reasons. The good news for Americans, though, is that we have enough colleges and universities to accommodate anyone who wants to learn, no matter what their high school record may be.
In my state, tests are pretty much exactly how you describe, because we have a relatively good education system. Students who come from our high schools can usually test out of almost a semester's worth of courses when they enter college.
In the state one of my friends is from, there's still controversy over teaching evolution and accepted cosmology theories, and many students need to take remedial courses on entering college because their high school education has so many gaps.
Just another example of how most guiding principles in US school systems are due to a lack of resources. We can only hope that some innovative people will be able to get around this (big advantage of US system is experimentation at the local level)
The United States spends more per student on education than any other country in the world, by a large margin. See http://www.oecd.org/education/skills-beyond-school/48630868...., page 206.
Lack of resources has never been a problem with the US education system, it's been the management of them.
Education, which people pay the state for, is failing in the USA. With 49% functional illiteracy, it is a catastrophic failure.
I find it very strange too big that doesn't mean it doesn't work. One of the interesting findings from the psychometrics of intelligence testing/g is that given sufficient motivation all tests are culture fair. You can work really hard to make a test that is approximately as fair to chimps as humans (Raven's Progressive Matrices) or you can decide on a corpus of culturally bound knowledge and publish the requirements and given sufficient preparation time the rank ordering will be approximately the same.
I'm presuming you're European as well, right? The truth is the SAT/GRE/GMAT/LSAT/MCAT are so highly g loaded that they're good IQ tests and our school leaving/college entrance exams are a combination IQ test and conscientiousness test. There will also be small effects for pitch perception in music but seriously the poor reliability of any one SAT question is irrelevant given enough of them. In the limit statistics don't lie. Not that there's anything wrong with complex summative exams, I'll support anything from economics quals to continental philosophy as the subject to be examined for rankings because for most purposes it's the rankings, not the substance that's important.
The method used here is not entirely problem-free either but it's a whole lot more practical than having a long string of multiple choice questions where you have a 25% chance by guessing and you get to know all the possible answers. This is not really representative of what you need in life or what you'll encounter in later studies in College/University.
Practicality is a false idol. Higher (and lower) education is more sorting/signalling than training. This can be blatant like Iran/Japan/Korea/USA where what university you got into is much more important than what you studied or what grade you got in the end, middling like the UK where there's a first tier (Oxbridge), a second (Russell Group) and also rans or other varieties like Germany (all universities are approximately equal, grades are approximately equivalent) or France (like Germany mostly with the grandes ecoles as very specialised high prestige schools). Some courses (ones that teach useful knowledge independent of ranking, like nursing or engineering) will have strangely higher salary averages for low prestige universities than one would expect, and the range for the comparable graduates of better universities will be oddly compressed.
Education is underrated, schooling the opposite.
Speaking out of fairly narrow self interest, the effect of the tests on my kids is that it dumbs down the curriculum, because drilling the test material becomes more important than moving on to more advanced topics or more complex types of assignments. They are both consistently in the 98th or 99th percentile on every test, and it seems like the testing is constant.
For my kids, more sophisticated and varied tests would mean a more sophisticated and varied curriculum.
This involves automated production of questions unique to a particular test-taker, and automated scoring of the answers.
Once that has been done, the difficulty of the test can even be adjusted on the fly to better determine the exact ability level of the student. Rather than having separate static tests for an expected range of ability levels, everything from preschool readiness to SAT and GRE equivalents can be tested by the same software.
If a child takes state-mandated standardized tests, and it shows a perfect score in math, the teachers may put that student in the most advanced math class taught at that grade level. But what if the test could show that the student's ability exceeded that of the typical student four or five years older, without becoming less precise for typical students the same age?
It's really a shame that standardized tests haven't yet evolved beyond multiple choice answers for human-written questions and scored by machines with optical scanners. But in general, I do think that test should be harder. They should be so hard that no one can achieve a perfect score, even by chance. And they should also be so easy that no one can score zero. It's the only way to get a good idea of the population distribution of the people you are testing.
They have somewhat, it's called adaptive, or "branched" testing. It is a computer-based test that gives you progressively harder questions to find the limit of your abilities and when you get them wrong it drops the difficulty again in an attempt to try to find out exactly where your knowledge drops off. That's an extremely oversimplified explanation, but you can read the Wikipedia article, which is OK:
https://en.wikipedia.org/wiki/Computerized_adaptive_testing
The problem is money, and to be honest, the motivations of who is buying the test (for some clients the testing is just meant to be a professional barrier to entry or to do the bare minimum to comply with a government requirement.) I worked in standardized testing for almost a decade on nearly every kind of test, helping psychometricians and test administrators, so I saw a lot of bad and good tests.
One wonders what a pure test for abilities would look like. The biggest factors in predicting long-term success are intelligence, knowledge, and conscientiousness. A hypothetical "how good you are at life" test could look something like this:
1. For intelligence, a straight-up IQ test. Raven's Progressive Matrices[1] is an excellent candidate. It can measure almost the entire human range of IQ, and it has little cultural bias. Heck, it doesn't even require literacy.
2. For knowledge, the format would be similar to existing tests, with one caveat: it must be very broad; so broad that no one person could know everything on it. To keep the length reasonable, one could use random sampling. Just as a vocabulary test need not ask the definition for every word in the dictionary, so too could a knowledge test remain accurate without asking for a copy of Wikipedia. To prevent cheating, it would be crucial to keep the exact questions secret each year.
3. Conscientiousness is the hardest to test, because it requires measuring one's ability to work hard over a long time. Besides self-reports, the only short test for conscientiousness involves telling the test taker that the task is unimportant. If they know the truth, the jig is up and they'll work hard.
This sort of test would be extremely hard to game. Training helps RPM very little. Studying for the knowledge section would help one's score, but it would have the side-effect of increasing one's overall knowledge. The biggest wrinkle is the conscientiousness evaluation. It either requires honesty on the part of the test-taker or trickery on the part of the test-giver.
While interesting to ponder, I doubt such a test will ever exist. It's not just students and parents who want to be able to game tests. School boards and other organizations want to be able to show improvement over time. If they can't teach the test, their true effectiveness (or lack thereof) might be revealed.
1. http://en.wikipedia.org/wiki/Raven's_Progressive_Matrices
I wouldn't dispute that those are important factors, but what about:
- initiative and/or risk-taking (which is very different from conscientiousness)
- attractiveness (either physical or non-physical things like charm or schmoozing)
- starting capital (it's easier turn $1 million into $2 million than starting with $0 and getting to $1 million)
- general health and fitness (successful people are rarely sickly or unfit)
- leadership
- number and quality of connections
Looking at the tech sector, it seems that a high level of intelligence, knowledge, and conscientiousness will get you a good job. But the most successful people in tech are not those with even higher levels of intelligence, knowledge, and conscientiousness -- for those people it's the other factors that dominate.
The fact that "C" is an insult is a sign of grade inflation, which might be another symptom of the bar being set too low (and that has to be compensated for by using grades like A+, etc. to distinguish the top performers).
When I was a professor I used to design my exams in two parts. The first were pure multiple choice questions that the students could game. The second were complex question that required integrating knowledge from multiple areas. Answering these question required understanding - these can't be gamed other than by learning.
You might be wondering why have the first part? The reason was because the administration set a quota on the pass rate. If I had only used questions of the second part then too many of my students would have failed. The bargain with my students was if you could fog a mirror you would pass, but to get an A you had to show understanding.