Hacker News Reader

Comment by marketer | Hacker News Reader

It is an unfair advantage, and it's frustrating to watch. For most of the answers, both Ken and Brad were trying to buzz in, but Watson always had better timing and buzzed in first. I'm sure that Jeopardy's buzzing system didn't take robots into account when it was designed, so it technically isn't against the rules. But it does give Watson a huge tactical advantage.

View on HN

I've been trying to think of a way to redesign the buzzer system to make it more fair, but I haven't been able to come up with anything. Tricky problem.

You could just give Watson a hard 200ms handicap.

So the game turns from "the absurdly smart computer stomps everyone" to "the absurdly smart (but deliberately crippled) computer may or may not stomp everyone, depending mostly on random chance."

That doesn't sound like a great improvement, imho.

What about just removing the timing element altogether, effectively making every round into (possibly iterated) Final Jeopardy?

That would give a more precise idea of just how many of the questions Watson was able to answer correctly, but at that point the discussion starts to feel like people proposing changes to the rules of chess in order to screw Deep Blue.

That would be more "fair," by which I mean that it would remove the obviously computer-favored dynamic of buzzer reflexes. Of course, it wouldn't make good TV, it wouldn't really be Jeopardy!, and I suspect Watson would lose. Against normal season contestants Watson would probably fair well (in a buzzerless competition), but against human champions Watson wouldn't stand a chance.

Do you know that? How? Stats?

There's no way to know what portion of the clues the human contestants knew the correct responses to. I've watched Ken and Brad play extensively though, so I very strongly suspect that they easily knew at least as many as Watson.

I would like some stats about that. As I said, Watson could have correctly answered 84% of the questions, nobody could answer 6% of the questions. That leaves a very thin margin for the humans, if past games are any indication one that is probably not large enough.

How to make the buzzers fair: If two people buzz in within time T of the question being read, randomly select who gets to answer. The value of T would be somewhere on the order of 200 ms, to distinguish between the two cases of "waiting until the question is finished before buzzing in" and "racing to figure out the answer the fastest".

It would probably be better to just linearly go through who gets to answer.

That wouldn’t be Jeopardy anymore. Timing matters. I like the 200ms solution.

They could revert to the old rule where you could buzz in as soon as you thought you knew the answer. It's hard to tell how long it takes Watson to come up with answers right now, but switching to the old rule would introduce an interesting dynamic where Watson would have to decide whether to spend more time chugging data or buzz in.

Also, this would be pretty artificial (no pun intended), but they could analyze previous all-human Jeopardy! episodes and figure out average buzzer response time, and perhaps incorporate that into Watson.

You could just change the game completely, and allow multiple people to answer simultaneously if they buzz in within a couple of seconds. It looked to me like Watson would be pretty competitive in that game, too.

I wonder when Watson gets the textual clue: the instant the clue appears on screen? The moment Alex finishes reading it? A midpoint? Is it trickled word-by-word at about Alex's speaking or average human sight-reading rates?

If Watson has confidence at the moment ringing-in is allowed, it seems it will always win. So how much time it has to achieve that confidence, as a function of how the text is fed, may be more important than buzzer mechanics.

Based on the same general idea of a common starting point that motivates waiting until after the entire question has been read before allowing any ringing in, I could see there being a tiny 'common period' where all buzzes are considered as coming in simultaneously, with the person chosen to answer then being chosen in round-robin fashion (or for maximal drama, favoring whoever is behind). After the tiny common period, it would be strictly based on first-to-buzz.

It would take away a twitch-timing factor that has been important for human champions, too, but offer more fairness with regard to computers and even people with slight ticks or timing problems.

The clue is given to Watson electronically the moment the clue appears on the screen.

While watching, I was actually hoping for really short questions which would cut down on Watson's time to process and possibly put Watson on more even footing with Jennings and Rutter.

Reply on news.ycombinator.com