Then even with all that debugging the model, and making architecture choices is a whole other thing he barely covers. Would love a good course on that.
If you learn the nuts and bolts you can point to any part of the transformer model and describe what numbers and operations are happening in the forward and backward pass. That sort of is the reality. Reading other people’s fuzzy explanations is probably like understanding quantum mechanics by reading quanta (you get the layman example but don’t really understand… btw I know little about QM!)
I needed to hop on other materials to exist. I van understand some of this but not all and running it and trying things out… probably not yet.
Well never for 100k context size as I don’t want to sell my house to pay for compute :-).
Don't listen to people telling you to learn the ML theory, if you don't know it, learn the functionality of the program. The one I recommended is one you can run on a normal computer.
…but you will be disappointed if you expect/hope to be able to recreate or modify things yourself.
You need a massive (multi TB) dataset of high quality data, and an array of 80GB graphics cards.
The barrier to entry isn’t knowledge; it’s money.
(So learning, yes. Try doing the fastai course. Change settings? No. Not unless you have a couple of million $$ in cloud credits to burn)
Like, even if some random bloke had thought of transformers on his own, he'd never be able to even test such a thing without having had unobtainable amounts of compute power and corpus input. As you said, it wouldn't even reveal its true potential until you're at some massive threshold of parameters, training time, etc.
The era of people like Huffman or Carmack or anyone "cracking" things independently seems impossible for the foreseeable future.
The things that AWS, Azure, OpenAI, and friends make available for a smol card swipe would literally break my brain when I was a bedroom hacker and my parents sunk 2 or 3 monthly salaries into a 166Mhz Pentium 1.
> The era of people like Huffman or Carmack or anyone "cracking" things independently seems impossible for the foreseeable future.
Wasn’t Huffman backed by a university? And didn’t Carmack do his best work when id software was printing so much money they literally didn’t know what to do with it all?
PS: many of the large datasets people use for these things are fairly standardized and keep showing up in paper after paper. I assume that means they’re available somewhere.
As for "cracking", people are still trying to make "the best game ever". This will never end ;)
I mean, in general yes, but in this specific example? Hmm…