Hacker News Reader

Comment by sharemywin | Hacker News Reader

Don't get me wrong it's pretty amazon what it can do but, I get the impression there's no confidence in the answers so I'm not really sure what to do with it. As with a lot of these systems how cherry picked were the answers

View on HN

some interesting stuff from one of the authors. could only solve a third of MIT undergrad problems but did well on other tests.

https://twitter.com/alewkowycz/status/1542559200051593219

Looks like Polish tests were included into training dataset.

From the paper, the Polish tests are from May 2022, and the training dataset was from January 2022, so this is unlikely.

Unless new tests consist of many questions similar to questions from the previous years.

I think this is huge issue with many current benchmarks. They need to create some handcrafted secret questions and never disclose them like it has been done with SquAD benchmark.

Students have the same advantage. What's your point?

First point is that LLM not necessary "solve", but potentially "memorize", while students can still be able to solve questions they didn't see.

Second point is proposal how benchmarks can be improved.

You can't "memorize" from previous tests, otherwise that's exactly what students would do.

LLM is much more capable in memorizing than human. Human (by my hypothesis) still better in logical reasoning.

You are mistaken, memorisation is trivial, what is hard is to recognise which of the memories to reuse later, and how to adapt it.

I would love to learn why do you think so.

It’s worth recalling that a structurally similar problem presented somewhere else on the internet could enter the training set. A non-trivial portion of problems likely just have variable names replaced.

If you can create a system that presents a limited number of novel solutions a human could double check each one.

Reply on news.ycombinator.com