Hacker News Reader

Comment by AuthAuth | Hacker News Reader

LLMs are beating math olympiads *after scraping the results from the competitions

How do you think LLMs know the competitions results before the competition takes place?

https://deepmind.google/discover/blog/advanced-version-of-ge...

A month ago LLMs got golden medal by competing at the same event as the students, on the same problems as the students, during the same period as the students.

Humans tend to have any chance of winning those also after scraping as much of previous results of those competition as they can. I don't think the most intelligent person that ever lived that haven't seen math could go and win math olimpiad.

Why are you acting like the digital equivalent of rote memorization is an impressive feat?

There are things that are impressive about generative AI, but getting correct questions on what is essentially a school test that is available online is not one of them.

Yeah its not impressive to serve questions to an AI that has the answers to those questions within its dataset.

Pick a single question from math olympiad. Solve it using any online resource beyond the solution to this specific problem itself, AI or service of smart people. Let us know how it went.

Why would they exclude the solution to the problem and give themselves a handicap that the AI does not have?

Because AI didn't have solutions for this year's problem in its training materials. Same way that students participating in this year's Math Olympiad didn't. Guys, don't you know anything about how such competitions work? You get limited time and a set of problems to solve. New problems, that weren't available anywhere before they were given to you.

That might be the issue. How can you appreciate what the AI achieved if you don't know anything about it beyond the name?

Per Gregor Dolinar, President of the IMO:

> “It is very exciting to see progress in the mathematical capabilities of AI models, but we would like to be clear that the IMO cannot validate the methods, including the amount of compute used or whether there was any human involvement, or whether the results can be reproduced. What we can say is that correct mathematical proofs, whether produced by the brightest students or AI models, are valid.”

Which is to say, they didn't give the AI model the answer, the AI model produced one without having seen it, and that it's valid. It might have taken extra time or had outside help, or otherwise cheated, but it didn't have the answer given to it to merely reproduce.

No such stipulation or claim is made. The only verified claim is that the proof is valid.

but also no stipulation otherwise is made by the contest runners. As the runners of a contest, I would presume they didn't give the answer to any of the test takers before they took the test because that would defeat the whole point of having the contest! If you think people/machines are cheating absent an explicit claim as such, I can't help you, but that seems unlikely to me.

It's entirely possible that both Google and OpenAI independently bribed a question writer for the answers and fed that into the LLM to be used as training data, and have not been discovered doing so, and that only then was the LLM able to generate the correct answer, but that seems a bit far fetched to me.

So earlier you said, AI answered with scraped answers. Since you now realize that was not the case, what's your belief now? That OpenAI (and Google) did a full on mechanical turk scam?

Because that's what the fragment you cited is saying. That MO can't assure that the answers provided by AI companies were really generated by AI (because they have no way of knowing that).

> Because that's what the fragment you cited is saying. That MO can't assure that the answers provided by AI companies were really generated by AI

It says nothing except that the proofs are valid and absolutely no assurance about them can be made. It does not tell you that the answers were not scraped.

I do not need anyone's assurance that if I'm gonna step out of the window, I'm gonna fall. Creating accurate (and predictive) model of reality in your head based on incomplete data is an essential skill. Your model is bad. Either ingest more data or train more on the data you already got.

I think you underestimate how much of what we call intelligence is just memorization of some basic principles and skills.

There is no “we”, here. What you and other AI hype proponents call intelligence and what I call intelligence are not the same thing.

Intelligence is defined as the ability to acquire, understand, and use knowledge in The American Heritage® Dictionary of the English Language, 5th Edition.

Based on that definition I think we are well on out way to building an artificial intelligence that is more capable than the overwhelming majority of the planet.

I don’t believe that you understand what an LLM is - that or you have a problem separating science fiction from reality.

You should let my employers know, they are paying my salary based on the assumption that I do.

And after all a company has never done anything foolish.

How much, do you think, of rote memorization competition level math is?

> There are things that are impressive about generative AI, but getting correct questions on what is essentially a school test that is available online is not one of them.

Nevermind. Try to solve one. You'll be enlightened.

I don’t need to try to solve one. The answers are available online. That is my entire point.

You need to solve one to put yourself in the shoes of the AI model to understand its achievement. When the models and students got the puzzles the answers weren't available online yet.

Reply on news.ycombinator.com