Hacker News Reader

> If you want to evaluate how good a teacher is, one of the things you need to measure is how well their students performed vs. how well those same students would have performed under the hypothetical average teacher.

But you can't measure that.

> it seems obvious that a set of students doing well does not, in isolation, indicate their teacher is a good teacher.

True. But in this case, at least, we're not talking about "in isolation". We're talking about a 17-year track record of a teacher's students doing well.

> what's a better way to do it?

When I went to school, the people who made these judgments were my parents. You can't make these judgments by formula, and you can't make them if you don't know the details of each individual case. To me, the fact that so many schools are fixated on "data-driven" student evaluations means that parents are not engaged.

People are not engaged. They are fixated on numbers, models, and abstractions, with the assumption that the numbers, models, and abstractions have actual meaning tied to them. They don't, aside from creating more students that create more systems that define more numbers, models, and abstractions. Some of these students get pissed off and try to create the opposite.

This is why the humanities are important. You can have the soundest logical systems, with the most elegant mathematical models, and completely miss the point. People get so caught up in the minutiae of measurements that they forget to see the big picture. Measurements don't mean anything when the measurements are used to measure themselves.

I swear, sometimes I really wonder whether society has it's head up it's ass. If these systems can model 'theoretical students' under 'better conditions', then why aren't the 'theoretical student' models running the world? Oh, that's right, because there's a gigantic difference between data and theory. How do they even know what a better student is? How can anyone in their right mind, define that? How can anyone even pretend to know what a great student is?

> When I went to school, the people who made these judgments were my parents. You can't make these judgments by formula, and you can't make them if you don't know the details of each individual case.

When I went to school, I used to think my grades meant something more than being a very complicated way of validating someone else's world view, in a way that tricks everyone into thinking we've made any progress at defining or understanding intelligence at all.

> To me, the fact that so many schools are fixated on "data-driven" student evaluations means that parents are not engaged.

The parents may be extremely engaged, but they just don't know whether to call their kid smart or not. If the kid complains about the book they read because it was boring, is that appropriate? If the kid programs a calculator to do 4 years of standardized test math homework, is that appropriate? Education is doing a fantastic job at driving the personality out of people by forcing them all to sit in the same box. The parents don't know, the teachers don't know, the government doesn't know, society doesn't know, yet we all pretend we know.

>"When I went to school, the people who made these judgments were my parents. You can't make these judgments by formula, and you can't make them if you don't know the details of each individual case. To me, the fact that so many schools are fixated on "data-driven" student evaluations means that parents are not engaged."

Many parents know their children are receiving a substandard (or even damaging) 'education'; the problem they often face is that they are powerless to fire the teachers, or switch schools. Being 'engaged' does basically nothing to fix the schools in these cases. Rich parents exercise school choice by moving to affluent communities with good schools, but the poor often do not have this option.

The VAMs aim to reliably give the school administrators access to the knowledge the parents (and often principals) usually already have, as well as give cause for discipline, incentive, or firing.

I could be interpreting your comment wrong, but it seems to put the onus heavily on the teacher, which seems a bit naive. My friend just started teaching in New York actually. And she specifically wanted to teach at a Title 1 school (i.e. poor) in order to reach students with less income and opportunity (she teaches an East Asian language in a predominantly non-Asian minority community). That was the idealized plan anyway. Yet, the school won't give her basic supplies. There is no lounge. No fridge. Ok, fine, that's tolerable (odd to me since I too was raised in an affluent community). But the teacher's need paper. There is a locked up supply room full of supplies, but the teachers are told they can't have access to it. Why? Nobody knows. Printing rights are curbed.

And on top of that all, NYC DoE requires a masters. That equals debt, if you went to a "good" school. But the DoE pays crap. Oh and for some reason at one point, the DoE lowered requirements to be a principal. 3 years of part time teaching, and you could be a principal. Her principal taught dancing for 3 years, and is now purportedly fit to analyze the effectiveness of foreign language teaching methods. But anyway, that's beside the point, that's just describing the environment.

Her program is new. But instead of asking students/parents and filtering for those who might be interested in an East Asian language, the administrators decided to randomly force students into the class, regardless of their prior language history or their year. You know what you get with that? Hostile students. The kind that scream fuck you in your face. OK, fine, just "standard" difficult students. But like any other job, if your "manager" has your back, you can usually deal. But this school doesn't believe in detention. Okay... The "dream" is that if a student is causing trouble, the deans will talk to said student. Uh oh, what wasn't accounted for was the saturation of the deans time due to trouble students. Now you got deans telling you that, sorry, they can't deal with disruptive students telling you to fuck off because they're overloaded. So now you have fire support and you're in the trenches alone. I'm not even going to get into the gay teacher's story. Once the students caught onto that...

This doesn't even include off-the-clock work that is required, which I won't get into. I'm sorry, but as someone in the tech field, or any privatized field, the shit that teachers have to put off with is insane.

I know teachers have been demonized, but the turnover rate in NYC for teachers is apparently extremely high (I have another friend working at the DoE itself). I'll have to get a source, but I seem to recall her saying it was around 70% after 2 years. And after hearing all the ridiculous war stories, I'm not surprised. Ha, another one of my teacher friends was moved to an empty classroom. Upon asking for desks, the administrators told her "we don't know where they are" and left it at that. So she had to essentially salvage desks marked for discard.

I'm not exactly sure how VAM works, but I'm skeptical that an algorithm can model something so complex.

> I'm not exactly sure how VAM works, but I'm skeptical that an algorithm can model something so complex.

And even if it could, should an algorithm like this be obscured from view? Should we rely on "black box" algorithms like this, or should we at least insist that, if not the code, the research behind the code be released openly so that it can be held to proper scrutiny?

> Many parents know their children are receiving a substandard (or even damaging) 'education'

Many parents believe that.

Many parents have children for which that is true.

The overlap between those two sets, OTOH, may be smaller than you think.

> the problem they often face is that they are powerless to fire the teachers, or switch schools. Being 'engaged' does basically nothing to fix the schools in these cases.

IME, this isn't really true -- but the perception it is true results in parents not being engaged. When parents are active in addressing perceived problems with teachers in public schools, it is very effective in getting those teachers out of the classroom. (And I've seen it happen numerous times both to bad teachers based on parents acting reasonably based on real problems, and to good teachers based on parents acting unreasonable out of offense that their special-snowflake children weren't getting handed grades on a silver platter. Sometimes it doesn't mean the teacher gets fired, sometimes they get laterally transferred or technically promoted to a position with the school system which is out of the classroom and not dealing with students, and sometimes they just voluntarily leave teaching. But parent activism is quite effective at getting teachers out of the classroom.)

  But you can't measure that.

If you have a model, such as:

score_at_end_of_year = score_at_start_of_year + a * parents_income + b * parents_education_level + c * teacher_skill + random_noise

You can figure out the values of a and b by comparing the performance of different students in the same class. If you have a large enough sample size to average out the random noise, you're left with (c * teacher_skill).

Not a perfect method, obviously - but if the best teachers at schools full of poor kids could get more incentive pay by switching to schools full of rich kids, would that be incentivising the right thing?

> If you have a model

Then you can obviously calculate what the model outputs for any set of inputs. But that's not measuring; that's modeling. They're not the same thing. You can't measure the students' performance with a hypothetical "average" teacher because that teacher does not exist. You can only model what you think are the relevant factors involved and then input actual data into the model to see what it says.

Can such models be informative? Certainly. Are they a substitute for human judgment? No way. Yet that is what these schools appear to be doing.

> if the best teachers at schools full of poor kids could get more incentive pay by switching to schools full of rich kids, would that be incentivising the right thing?

Does that actually happen? And if it does, would this model prevent it? Once again, models are no substitute for human judgment.

> that teacher does not exist

The proposed model makes no assumptions about the "average" teacher -- it proposes a simple metric applied to student performance. Under this metric, some teachers would get a positive score because their students' scores improved more than the average. Others would get a negative score because their students' scores improved less than the average.

> would this model prevent it?

I don't know, but the goal of the model is to prevent this, or even reverse it -- that is, the best teachers would want to work with the least advantaged kids, because arbitrage -- it should be easier to make a difference for kids starting at a low performance level, and harder to make a similar difference with kids starting at a higher level.

I don't know whether this works even a little bit in practice, but the thinking sounds promising.

> The proposed model makes no assumptions about the "average" teacher

Sure it does. How else is the teacher_skill rating calibrated?

> the best teachers would want to work with the least advantaged kids

Why is this necessarily a good thing? Shouldn't there also be an incentive to have the best teachers work with the brightest students, to ensure that those students are actually challenged instead of just skating through school?

Put another way: such modeling is probably informative over the aggregate. That is, when comparing school districts to each other. But such modeling is likely not valid for an individual teacher.

Yes, and then you can figure out different values of a and b for a different teacher, different class, different year, different town, etc.

And you're really left with c*teacher_skill + noise, where the noise in the teacher_skill measure may be what results in some good teachers getting bad ratings.

In my view, empirical modeling assumes that there is some underlying regularity that can be modeled, but this assumption seems questionable. And part of the process has to be to decide an acceptable level of good teachers getting bad ratings.

Most HN readers are programmers. Just imagine this logic being applied to yourselves. Your performance is measured in a single week at the end of each year by statistically-controlled standardized tasks. Your manager's performance is then measured by the statistically-inferred effect that working for her has had on your performance given your previous year's performance and a model of how programmers' performance changes over time based on a few variables.

How well do you think that would work? Would it accurately reflect the work you had done in the previous year? Would it generate the right incentives for you to actually do good work and for your manager to help you grow?

Or is it more likely that random variation would dominate meaningful variation. That getting sick or feeling depressed that week would have markedly more impact than all the ways you really got better or worse that year? Or that it would generate perverse incentives to, say, learn about the standard task early, to prepare for the kinds of tasks you were measured on instead of the ones that produce value?

Or, for that matter, that it would destroy the motivation of everyone involved by replacing their intrinsic motivation to do well with an extrinsic motivation to get rewarded?

To be clear, most HN readers' employment and compensation is based on a manger's holistic sense of that HNer's value. We can generally be fired without cause.

If teachers want to argue that they should be in an employment regime similar to programmers, sure, I'm sympathetic. But my strong impression is that they don't want that at all.

> Your performance is measured in a single week at the end of each year by statistically-controlled standardized tasks.

My main point was that if you're going to appraise teachers based on their students' standardized test scores, it would seem to make sense to use a statistical model that tries to measure how much of an effect the teacher has on the predicted test scores of their students. I'm completely open to the argument that we shouldn't be appraising teachers based on student test scores, but that does not seem to be the position the article is advocating.

If you don't have some sort of system to extrinsically reward people with different skill levels different amounts than the best people are eventually going to figure that out and go do something else where they'll get paid what they're worth.

If you don't believe this think about how pissed you'd be if that idiot down the hall who couldn't write good code to save his life made as much money as you do.

You seem to assume that money is the sole motivator, everywhere.

What's really going on is that you're trapped in your own world view, validated by people like you, all alike in your social circles.

My advice: at least try and get out of it a little. I'm up to my gills in the high-tech culture, working at a startup in the Silicon Valley - but my wife is a teacher. I went to a Halloween party hosted by some of her co-workers - it's a tremendous difference in terms of culture, ideals, etc. It was literally like meeting people from a different country.

Metrics aren't the final answer to every question. When dealing with people (which is what education is doing), a simple, single, narrow number used to reduce everything to a handy metric can be incredibly misleading.

Whenever I head some folks saying that we need more metrics and more competition to make the education system better, I just want to tell them: "thank you, but sit down and shut up, cause you've no idea what you're talking about."

My wife is also a teacher(1). While I didn't go to a Halloween party with her coworkers this year I've been to quite a few other parties with teachers. So maybe don't make assumptions about my world view huh?

I never said that metrics were the final answer to every question either. Teaching is obviously an incredibly nuanced profession. If, as Michael Lewis showed us, we can't even measure baseball player's performance very well then oh gosh it's going to be much much harder for teachers (or programmers too!).

But that doesn't mean there aren't any differences in skill either. There are great programmers. There are not so good programmers. There are great teachers. There are bad teachers. If we deny this reality and just pay them the same amount then eventually the good ones are going to figure out they're getting screwed no matter how much they intrinsically like the gig.

1. http://www.fordham.edu/academics/programs_at_fordham_/commun...

> If you want to evaluate how good a teacher is, one of the things you need to measure is how well their students performed vs. how well those same students would have performed under the hypothetical average teacher.

But underlying this assumption is that you have a good measure of student performance in the first place -- or that such a measure can easily be crafted. It's questionable if it even makes sense to quantify a student's abilities into a single catch-all number.

But even if quantification of student performance makes sense, it becomes problematic when it becomes high-stakes (i.e. teachers' payment or school funding depends upon test scores). The problem with tests (or any other metric tied to incentives) is that they are gamed over time; SATs become increasingly meaningless with the cottage industry of test-prep; and so do standardized tests become increasingly meaningless as teachers are incentivized to teach directly to them. (In politics, crime rates are gamed, jobs rates are gamed, etc.).

Were schools so terrible before standardized tests -- when administrators and other educational experts rated teachers? There were existing non-mechanical rating systems that of course predated standardized testing -- is it necessary to mechanize evaluations simply because we like the superficial illusion of objectivity, even if that objectivity bears no resemblance to the thing we are trying to measure?

Or even acknowledging that perhaps existing non-mechanized teacher performance measures may have been flawed, have standardized tests somehow improved schools? No, probably not -- teachers and students both are more miserable and are increasingly tied down by the requirements of possibly meaningless tests.

> The particular VAM system used to estimate this metric may be flawed or even completely broken. But at the point in the article this pull quote is located, that argument had not been made.

> My biggest problem with the article is it doesn't describe alternate methods of teacher evaluation. VAM may be flawed, but how does it compare to the other methods? If we accept that teacher evaluation based on student performance is necessary (is it? I have no idea!), what's a better way to do it?

The main point I took from the article is that the VAM approach is a black box. It is data driven and we have some reason to suspect it should be useful, but we simply have no way to audit that on specific cases.

Combine that with the fact that this particular teacher is exemplary, yet got an abysmal score, we are faced with the worst possible result: Someone that should get a great score received a terrible one, and we don't know why. So we can't figure out what went wrong or how to fix it.

This isn't a new issue. Statistical models used to be more human-understandable, then over several decades neural networks and big data and so forth have become popular. They often achieve good results, but often remain partially or entirely black boxes.

It makes sense to use a black box method to recognize handwriting, for example, if it does well on average. It's fine if it messes up on some samples. But teachers deserve to be treated fairly, each and every single one.

That's the tension here - better methods on average tend to be less auditable on individual cases.

Think about it this way, using your own claim: if VAM is measuring the amount of value a teacher adds to student test performance based on an average student of similar background, and the average student in a wealthy neighborhood does well regardless of the teacher, then every teacher in every wealthy neighborhood adds no value! Does that justify firing/docking the pay of all your teachers? Obviously not. It says that the kind of value a teacher adds necessarily depends on the community. It's a qualitative difference, not a quantitative one.

> if VAM is measuring the amount of value a teacher adds to student test performance based on an average student of similar background, and the average student in a wealthy neighborhood does well regardless of the teacher, then every teacher in every wealthy neighborhood adds no value!

I'm no statistician, but I don't think this is actually true. We would expect students who are likely to do well on standardized tests under a Hypothetical Average Teacher (HAT) to benefit relatively less from a good teacher than students who are likely to do poorly on standardized tests under a HAT - at least when we use standardized test results as the metric. But presumably, any sophisticated statistical model will take this into account. You would simply adjust the model such than a unit increase in actual score vs. predicted score is "worth more" in terms of indicated teacher performance as predicted score increases. It's just a case of diminishing marginal returns, and not unique to teaching or education.

It might be harder to distinguish signal from noise as the predicted test score increases. But it still seems a better system than "the students have done well, therefore the teacher has done well".

My point is that as wealth and average test scores increase, certain kinds of value become more important in distinguishing teachers. Values which VAM does not even attempt to measure.

>"if VAM is measuring the amount of value a teacher adds to student test performance based on an average student of similar background, and the average student in a wealthy neighborhood does well regardless of the teacher, then every teacher in every wealthy neighborhood adds no value"

If you are going to make arguments against VAM, please come up with some stronger ones. Applying statistical controls for student test results in situations like these is trivial, and teachers do make a big difference. If the teachers really made no difference, why not just hire the cheapest labor available, and spend the rest of the money on extra-curricular programs?

How is that? In some communities the difference could be between learning to read and finding an appreciation for modernist poetry. Or more drastically, the difference between instilling a sense of civic duty and keeping kids out of prison. We do expect teachers to have a part in all of these things.

Speaking as someone who reads a book per week, and has spent all their life in affluent communities with highly rated educational institutions; I can confidently say that for the overwhelming majority of public schools (even those in rich suburbs), achieving functional literacy with some comprehension and analytic ability for the majority of students is a lofty goal. I do not know of any public school where the difference between a good teacher and a bad one is "between learning to read and finding an appreciation for modernist poetry"; maybe this will be a problem in the future, but it is not something we should be grappling with now.

I am not under the impression that what "keep[s] kids out of prison" is "a sense of civic duty". Good career prospects keep people out of prison; high conscientiousness may be correlated with low rates of imprisonment, but conscientiousness is also a luxury the desperate can ill afford to indulge in.

But is it done?

In NFL fandom, Football Outsiders's Defense-adjusted Value Over Average (DVOA), is perhaps the best-regarded predictive model of results. In a shoot-out on /r/nfl last year, it outdid ~20 other ratings and rankings in predicting outcomes. One can only imagine that if predicting NFL games were as politicized as school outcomes, there'd be a powerful lobby claiming DVOA "doesn't make a lot of sense".

You're right, this particular model might be bad. But something along these lines will be good, inasmuch as correctly approximately evaluating a teacher's effect on outcomes is good. Science always wins.

It should be simple to develop a predictive model of programmer effectiveness. We can measure the number of lines of codes, number of defects committed, number of defects fixed, etc. against the average developer on the average software project and then we have a way of approximately evaluating a programmer's contributions to a software project.

Science is really hard.

See, the thing is, people are already trying really hard to do that. Practicing programmers, even.

And it's hard to do that even when the desired outcome is just good code.

Now try and do it when the desired outcome is good people (education).

Good point. Would you agree that: even if a phenomenon (for example performance of some sort of worker) is hard to model, we should still try, especially if that phenomenon provides enormous value to society?

Life is a hell of a lot more complicated than the game of football.

The people advocating for a metric ought to be the ones who are expected to establish that they are sensible. Nonetheless, education "reformers" have consistently been promoting metrics with enormous variation in ranking.

Of course, if your purpose is to garner the political support of people who say "it sounds like it makes absolute sense" actually measuring meaningful things is irrelevant.

> well those same students would have performed under the hypothetical average teacher.

I don't understand how do you do that. I can see this hypothetical setup -- you start a genetic experiment, clone all the student's DNA. Produce the same exact # of babies. Raise them the same way, respectively as each of the original students. Then you build a robotic teacher, whose skills, personality, experience, represents the average of all the personalities, experiences and skills of all the teachers in the state (luckily you just do this once per year, and then make lots of clones, which should be easy).

Then you give the group of clones to this "average" robot teacher and you see how well the students do. (It is optional to keep the clones after this experiment, they could be used for spare parts for later maybe...).

So now you have your VAM measure and you can assign a score of 1 to 20 to this teacher.

That was all sarcastic of course. But ok, how does this model work? Can you explain.

> I don't understand how do you do that

Here's a simple way to compare someone to the average teacher.

Two tests. One at the beginning of the year, one at the end. Take the average improvement across the school, across the district, across the state. Now apply that average school-wide improvement to a classroom's original scores in the first test to see how those students should have performed on the second test. Compare that number to how they actually performed.

I came up with this in about 10 seconds, there's obviously a better way to do things, but comparing someone to the average teacher really isn't that hard to do.

The most important part is coming up with a test that actually measures understanding. Open ended questions are usually a good place to start. Many tests I have taken really require you to understand the material.

> Two tests. One at the beginning of the year, one at the end.

Is it the same test (same questions) or a different test? I would guess same type of questions.

Also what if the students are already top performing and don't improve but actually get a slightly worse. (As in they all get 95% and then, well a few get sick and skip a few months from school and now the class gets 94%).

I guess I am still confused at how average expected improvement is supposed to work reliably across all those population models.

Do we assume the school-wide population of students is uniform enough to calculate a meanigful average improvment from it which can be applied very classroom individually.

There are a lot of school districts that have very mixed populations (economic, ethnic, cultural) backgrounds.

Same about the state. Some states districts that maybe very rural and undeveloped mixed in with a metropolitan area some place across the state.

All great questions, and all things I think can be solved.

What it comes down to is, what's our other option? Does it even matter when it's nearly impossible to fire a bad teacher?