Hacker News Reader

Comment by wrs | Hacker News Reader

I would be interested in hearing the results from humans presented with adversarial answer options. You may say that a machine learning correlations between words isn’t really learning science, but I wonder how many human students aren’t either, just pretty much learning correlations between words to pass tests...

View on HN

They do give an example of a question, in which the model chose an incorrect answer in the adversarial setting:

"The condition of the air outdoors at a certain time ofday is known as (A) friction (B) light (C) force (D)weather[correct](Q) joule (R) gradient[selected](S)trench (T) add heat"

I assume this might be characteristic for other questions as well, although I don't know anything about the Regents Science Exam and whether there are multiple questions about closely related topics.

That’s a terribly worded question anyway. Of the original answers, ‘weather’ is the least worst but it’s still vague.

It is a well-worded question for its purpose. The whole point is that, of all the options given, only one is justifiable (and it does not require a tendentious stretch to justify it, either.) Even “light” (which was not chosen) only applies half the time, on average. This is a valid test of natural language understanding.

Remember when IBM went on Jeopardy? There was a question about which Egyptian pharaoh. A human with some knowledge of history might mix up Ramses and Seti, or whatever, or just not know the answer, but know that they didn't know. Watson answered "What are trousers?" with supreme confidence.

Jeopardy is fun and games and it was great for the blooper reel, but they're trying to sell this stuff to diagnose cancer and guide police efforts. Failure modes are kind of important.

Most multiple choice math problems can be completely circumvented by simply finding the digital root of the expression in the problem. I was surprised to find this to be true, even on college entrance exams.

Seriously? Can you explain the mechanism? 'cos that sounds like - well, numerology, to be honest.

The digital root of the problem expression will be the digital root of the answer, because the answer IS the problem expression. They're the same number, just with all the pieces jumbled up. Comparing against digital roots of answer choices will usually eliminate most of the wrong answers. Any remaining wrong answers tend to be obviously wrong. 2 + 2 != 13 obviously.

Can you give an example? I am intrigued, although suspect you are making a joke!

You can check the properties of digital roots on wikipedia:

https://en.wikipedia.org/wiki/Digital_root#Properties

Example from ASVAB practice math test:

(x+4)(x+4) =

A. x^2+16x+8

B. x^2+16x+16

C. x^2+8x+16

D. x^2+8x+8

Since we don't have to solve for X in this problem, we can just assume x is 1, which would make the digital root of the problem expression 7. Assuming x = 1, the digital roots of A, B, C and D are 7, 6, 7 and 8 respectively. C is the answer because its last term is the square of 4.

Perhaps I am being dumb, but I don't see what the digital root brings to this.

You can do the same thing without the digital root. substitute 1 into all the possible answers gives: A: 25 B: 33 C: 25 D: 17

Substitute 1 into the question gives 25. Therefore only A and C are possible answers and it must be C because 4*4 = 16

This is the same process as your answer but without the digital root!

Agreed. The real trick here is knowing that, since we don't need to find x, we can sub in an arbitrary x and do the comparison numerically instead of algebraically.

Of course there's a tiny chance that one of the other equations would be equal at your arbitrary x value. If so and you get two 'right' answers just try again with a different x.

It is the DeMillo-Lipton-Schwartz–Zippel lemma

If two polynomials evaluate to equal numbers with random variables values, the polynomials are almost certainly equal themselves

Ofc, x = 0 and x = 1 are hardly random

I suppose such is possible without digital roots. Let's try the same problem, but this time we'll multiply our constants by 3,571:

(x + 14284) (x + 14284)=

A: x^2 + 204032656x + 28568

B: x^2 + 204032656x + 204032656

C: x^2 + 28568x + 204032656

D: x^2 + 204032656x + 204032656

The digital root of the problem expression is 4. The digital roots of A, B, C and D are 4, 6, 4 and 3 respectively. The digital root of the square of 14284 is 1, and the digital roots of the last term of A and C are 2 and 1, respectively. The answer is C.

Of course, it is quicker just to multiply and add the last digit of the constant to find the only possible answer (or count digits to get an order of magnitude estimate, which will also show C as the only possible answer).

When a problem of this form uses big enough numbers to warrant a shortcut, digital root is almost always a suboptimal shortcut.

Yes you are the heuristics master!

Maybe is case of large numbers you can compute digital root easier than the answer.

For example, 25 * 26 = 650, digital root 2. Or take digital root of 25 and 26 first, get 7 * 8 = 56, same digital root 2.

I can't decide if I've just seen magic, or conjuring. I'm gonna have to go away and think about this.

This is true, but in many cases amounts to reducing the problem to a more time consuming problem, though one that requires less knowledge. On timed tests, this can be a big negative.

Reply on news.ycombinator.com