Hacker News Reader

Comment by ftxbro | Hacker News Reader

it doesn't matter, the 'bitter lesson' as coined by Rich Sutton is that stacking more layers with more parameters and compute and dataset size is going to swamp any kind of clever 'feature engineering' like trying to be clever about phonetic tokens. Karpathy for example just wants to go back to byte tokens.

View on HN

If we don't fix up issues caused by the tokenizers, than techniques which literally remove superfluous computation (i.e. through filters of the LLM probability distribution) are useful as a stop-gap.

Switching to bytes is the ultimate fix, but for the interim, if you want reliable rhyming with an LLM, you need filter-assisted decoding: https://paperswithcode.com/paper/most-language-models-can-be... and replicas post about this work: https://replicate.com/blog/turn-your-llm-into-a-poet

Yes, but how much extra layers and computing power do you need? Of course, phonetic tokens are awkward idea, but there is a reason why word "human" is encoded as only one token.

I don't think that is intuitive at all. "Clever feature engineering" like trying to create columns from calculations of tabular data, sure. You're not going to move the needle. But the basic representation of unstructured data like text could very believably alter the need for parameters, layers, and calculation speed by orders of magnitude.

> "I don't think that is intuitive at all."

That's exactly the point. Every intuition is always on the side of feature engineering.

Not really.

You would be wrong at the scales we are talking about.

The whole point is that it is unintuitive.

If you replace a tokenizer with 5 bytes per token on average by a byte-level representation, you now need 5 times as much memory and (depending on the specifics of the attention mechanism) 11 to 25 times as much compute.

At the scales we're talking about, that's quite a hefty price to pay, and it doesn't even take into account that you might need more layers to replace the processing that was implicitly done by the tokenizer.

Most current implementations can't count syllables at all, so it would get you at least that far.

I think that’s part of the reason I would love to see somebody try it, because intuitively I think it would make a difference, but it may not.

To me it’s like changing the periodic table, at the macroscopic scale it may or may not make a difference.

> I think that’s part of the reason I would love to see somebody try it, because intuitively I think it would make a difference, but it may not.

The crazy thing is it's already solved. YC should just spin a subreddit.ycombinator.com for each one, there's a nice search that works and some very nice apps for reading. What the reddit shareholders are about to buy is incredibly fragile and the management is fucking with it so much it's obvious they only care about the payola.

Reply on news.ycombinator.com