Comment by albert_e | Hacker News Reader

Comment by albert_e | Hacker News Reader

OT: What is a good place to discuss the original video -- once it has dropped out of the HN front-page?

I am going through the video myself -- roughly halfway through -- and have a fw things to bring up.

Here they are now that we have a fresh opportunity to discuss:

1 - MATH and LLMs

I am curious why many of the examples Andrej chose to pose to the LLM were "computational" questions -- for instance "what is 2+2" or some numerical puzzles that needed algebraic thinking and then some addition/subtraction/multiplication (example at 1:50 mins about buying Apples and Oranges).

I can understand these abilities of LLMs are becoming powerful and useful too -- but in my mind these are not the "basic" abilities of a next token predictor.

I would have appreciated a more clear distinction of prompts that showcase core LLM ability -- to generate text that is acceptable as generally grammatically correct, based in facts and context, without necessarily needing the ability of a working memory / assigning values to algebraic variables / doing arithmetic etc.

If there are any good references to discussion on the mathematical abilities of LLMs and the wisdom of trying to make them do math -- versus simply recognizing when a math is needed and generating the necessary python/expressions and let the tools handle it.

2 - META

While Andrej briefly acknowledges the "meta" situation where LLMs are being used to create training data for the training of and judge the outputs of newer LLMs ... there is not much discussion on that here.

There are just many more examples of how LLMs are used to prepare mitigations for hallucinations by preparing Q&A training sets with "correct" answers etc

I am curious to know more about the limitations / perils of using LLMs to train/evaluate other LLMs.

I kind of feel that this is a bit like the Manhattan project and atomic weapons -- in that early results and advances are being looped back immediately into the development of more powerful technology. (A smaller fission charge at the core of a larger fusion weapon -- to be very loose with analogies)

<I am sure I will have a few more questions as I go through the rets of the video and digest it>

Regarding 1 - MATH:

Somewhere in the video he says that LLMs have expert (only slightly fuzzy) knowledge about a lot of topics, but fail with simple math questions. Many non-technical people anthropomorphize LLMs and don't know that they can't think or calculate like a real calculator. LLMs compute tokens and you can improve the performance, if you don't put too much computation into a single result token.

I think it's an excellent example to show the capabilities and limits of LLMs. For softer topics, you can argue a lot more about what's considered to be right or wrong. With Math, you have a single correct answer that can be evaluated and people assume that computers are good at computer things, such as calculating numbers, even though LLMs actually aren't good at this.

The takeaway is: Prompting and "computational complexity per token" matter and if you understand how it works for math, you probably understand how it works for softer things like answers about law or whatever.

I've definitely done things like "give me a time stamp" then took too long to realize the time it gave made no sense. You get used to it working well when it does, and then it doesn't, and it's hard to switch the skepticism back on in response.

I believe Andrej Karpathy runs a discord, which is linked on his website [1]. I haven't participated personally, but from what I've seen, it's very active.

[1]: https://karpathy.ai/zero-to-hero.html

> for instance "what is 2+2" or some numerical puzzles that needed algebraic thinking

there is only one algebraic approach to solving something like 2+2 and that is counting! 2+2 = (((0 + 1) + 1) + 1) + 1). but llms are infamously bad at counting. which is why 2+2 isn't an algebraic problem to an llm. it's pattern matching or linguistic reasoning token by token.

LLMs are bad at counting because nobody counts in text, we count in our heads which is not in the training material.

i don't think it's just about the training material. it's also about keeping track of the precise number of tokens. you'd have to have dedicated tokens for 1+1+1+1 and another one for 1+1+1+1+1 etc.

Internal representation is multidimensional vectors. A typical 4096 in q4 one can name every particle in the universe and have over 4000 dimensions left for other purposes

i don't think that is a valid argument.

This result --

https://x.com/yuntiandeng/status/1889704768135905332

Is this a consequence of the fact that "multiplication tables" For kindergarteners are available online (in training data) abundantly ... typically up to 12 times or 13 times table as plain text ?

For point 1, he gets into that more later in the video - e.g. specifically on counting, and about how/when to have models invoke tools instead of doing math themselves, etc.

Also for the second point, check later in the video when he talks about RL and (simulated) RLHF - he gets into the feedback loops of models training each other and the collapse that follows.

> I am curious to know more about the limitations / perils of using LLMs to train/evaluate other LLMs.

At the extreme, there is this paper on ‘inbred LLMs’: https://www.nature.com/articles/s41586-024-07566-y

> I am curious to know more about the limitations / perils of using LLMs to train/evaluate other LLMs.

the entropy goes up in such case (up means less information). The result will be as if someone recompressed mpeg with another lossy compression. You can sometimes see the results on the internet.

Reply on news.ycombinator.com