Hacker News Reader

Comment by buildbot | Hacker News Reader

Why only MNIST and a Graph CNN? Those are small and somewhat odd choices. Scale these days should be at least 100 million param models and something like OpenWebText as a dataset in my opinion. Not sure what the SoTA is for visionm but same argument there.

View on HN

This paper is from a small group at an academic institution. They are trying to innovate in the idea space and are probably quite compute constrained. But for proving ideas smaller problems can make easier analysis even leaving aside compute resources. Not all research can jump straight to SOTA applications. It looks quite interesting, and I wouldn't be surprised to see it applied soon to larger problems.

I’ve been in a small group at an academic institution. With our meager resources we trained larger models than this on many different vision problems. I personally train LLMs on OpenWebText than this using a few 4090s (not work related). Is that too much for a small group?

MNIST is solvable using two pixels. It shouldn’t be one of two benchmarks in a paper, again just in my opinion. It’s useful for debugging only.

Again, a small academic institution may not have the experience or know-how to know these things.

I thought so at first, but the repo's[0] owner and the first name listed in the article has Seoul National University on their Github profile. Far away from a small academic institution.

[0]: https://github.com/ironjr/grokfast

> MNIST is solvable using two pixels.

really? do you have any details?

agree it has no business being in a modern paper

It's a free world. Nothing stops you from applying their findings to bigger datasets. It would be a valuable contribution.

How can MNIST be solved using just two binary pixels when there's 10 classes, 0-9?

i'm also curious but my understanding was MNIST pixels are not binary due to some postprocessing artifacts

Oh hm, so they are. I thought they were binary because they used a digital pen to create them, IIRC, and logistic regression is always the baseline; but checking, they technically are grayscale and people don't always binarize them. So I guess information-theoretically, if they are 0-255 valued, then 2 pixels could potentially let you classify pretty well if sufficiently pathological.

> They are trying to innovate in the idea space and are probably quite compute constrained.

Training a GPT-2 sized model costs ~$20 nowadays in respect to compute: https://github.com/karpathy/llm.c/discussions/481

$20 per attempt. A paper typically comes after trying hundreds of things. That said, the final version of your idea could certainly try it.

Baseline time to grok something looks to be around 1000x normal training time so make that $20k per attempt. Probably takes a while too. Their headline number (50x faster than baseline, $400) looks pretty doable if you can make grokking happen reliably at that speed.

Grokking may not even occur for datasets of that scale. Even the MNIST experiments require dropping the training data size from 50k examples to 1k. The reason for this is that the phenomenon seems to occur at a critical zone of having just barely enough training data to make generalization possible. See https://arxiv.org/abs/2205.10343 for details.

Even figuring out how to induce grokking behavior on a 100M model or OpenWebText would be a big leap in the understanding of grokking. It's perfectly reasonable for a paper like this to show results on the standard tasks for which grokking has already been characterized.

It’s because that’s where the effect is showing upright now. This is the situation where the analogy to pre-paradigmatic optics is pretty strong. If you telescope to take pictures of Jupiter was having problems with rainbow fringes, so you designed the defraction grading to investigate the fringes

Several reasons - computational effort, effort it takes to reproduce results on complex datasets, academic publishing model.

Academic research is often times about making incremental steps and limiting uncertainty.

Making something work for MNIST is already so much work, researchers don’t have the time, money, and energy to run experiments for 10 datasets.

Complex datasets are much harder to get a proper model trained on due to increased complexity - larger images, tasks, classes, etc.

Also, as soon as you run your experiments on more datasets, you create an opportunity for reviewers to take you down - “why didn’t you test it on this other dataset?”

i don't think grokking has been demonstrated in a large model yet

There's a recent paper here where they've seen in a deeper model.

https://arxiv.org/abs/2405.19454

It's a bit surprising that apparently this hasn't been done before - to see how universal of a phenomenon this is.

Reply on news.ycombinator.com