Relevant tweet (imo) from Andrej Karpathy - https://twitter.com/karpathy/status/1657949234535211009
What are the leading alternative approaches? Character-by-character prediction?
Thinking about human brains, it seems we operate on an ever-evolving set of symbols, at various levels of abstraction. I believe LLMs capture higher level symbols in their weights, but would it make sense to try to include a dynamic, self-updating set of symbols into the “what is being predicted” part of the training?
That being said, didn’t the first gpt-like models do character level prediction? Essentially no tokenization?
Also, the idea of “what if we removed / rethought tokenization” is in no way equivalent to “what if we didn’t even use neural nets”.
Throwing in wild ideas maybe works when you know what you're talking about and want to get people to think ouside the box, but does not work when you don't have a clue and expect people to explain things to you and to find a solution to your wild ideas.
Re: removing tokenization. Tell me how you would remove tokenization and keep NN. Go ahead.
Your incredulity and exasperation is misplaced. If you can’t realize that then I think we are done here.
Btw, since we are already in the thick of things… Even if you were 100% correct, which you aren’t, there is really no need to be rude. It just makes things worse.
> Re: removing tokenization. Tell me how you would remove tokenization and keep NN. Go ahead.
You can make a language model using character-by-character prediction, but I believe they don’t perform as well. This is now my 3rd time mentioning this.
No tokens is like no compression. It's much less efficient. My previous sentence is 12 tokens (24 bytes), or 58 characters (58 bytes). Your self attention block will be able to handle a much smaller effective context.
Also, especially in english, tokens by itself have quite a lot of semantic meaning by themselves, while characters are basically meaningless. So it probably needs extra work to piece those characters together vs operating on a token that already has meaning.
maybe one approach would be to tokenize not on tokens automatically found using BPE (Byte Pair Encoding) but tokens that are semantically meaningful - word prefixes, suffixes and syllables. Of course we will have to decide whether to divide the word helicopter into heli-copter (pronunciation) or helico-pter (meaning), but both alternatives encode much more transferable meaning than GPT's he-lic-opter.
Of course the difficulty is that that's ... difficult. Your tokenizer would first need to understand what language a word is in, then how to split it in the context of that language. Maybe a task for a whole ML model, where current tokenizers can be simple largest-substring matchers.
Kind of a recursive, self-evolving tokenization process?
I’m pretty new to deep learning, so this analogy might be off. But I’m reminded of convolutional layers inside image-based models. Could we have “tokenization layers” in a GPT model?
---
Edit: I asked GPT-4 about my idea.
It said it the comparison between tokenization and convoluational feature detection is a “someawhat accurate” analogy. And it agreed that making the encoder a fixed separate process that doesn’t evolve during training does limit the GPT model in certain ways and introduce quirks.
But it said that it might increase the computational requirements significantly, and that the transformer-architecture doesn’t lend itself to having “tokenization layers”, and it isn’t clear how one could do that.
It did say that there may be ways to work around the main challenges, and that there is some research in this direction.
That's not "meaning", that's etymology. If you divide it by "meaning" you get heli-copter, which is evidenced by the way the speakers use those morphemes: helipad, heliskiing, heliport,... and parcelcopter, gyrocopter, quadcopter,...
In a sense, none, character (with whatever encoding scheme you want) models have been done. OTOH, you get effectively a smaller context window (since tokens represent one or more characters), and since tokens are semantic unions you push more work into the main network to acheive similar quality.
> Thinking about human brains, it seems we operate on an ever-evolving set of symbols, at various levels of abstraction.
Analogies between human brains and LLMs are often more obscuring than illuminating, but I think if you want to go down that route here, that’s equivalent to doing continuous fine tuning on both the tokenizer (that takes sensory input and maps to semantically meaningful symbols) and the main model.
> I believe LLMs capture higher level symbols in their weights, but would it make sense to try to include a dynamic, self-updating set of symbols into the “what is being predicted” part of the training?
“Dynamic, self-updating” simply isn’t part of the way modern LLMs work, independent of whether you are looking at the tokenizer or main model. If someone could make it practical, sure, there are lots of things it would potentially improve.
I’m curious about the latest research related to continuously improving the tokenizer while training the main model (or something equivalent).
Edit: the token list for GPT-4 looks pretty clean. Overwhelmingly dominated by code but just eyeballing fragments of the set I didn't see any tokens in there that were obviously going to be rare.