eldenring

No their claim is that there are dimishing returns for a fixed compute budget (in training) to scaling up data past that threshold vs. scaling up params.

This doesn't take inference into account either, obviously.

Language models are a path to Super Intelligence. Not the most efficient one, but it might be good enough to be able to brute force it.

You can say the same thing about RNNs. Technically nothing is turing complete without infinite scratch space.

Fine-tuning an entire language model to solve this problem is like using a sledgehammer on a nail. We have had tools for this for years, for example just label some data and train an SVM on your embedding space for classification.

> Yes, it lets you shoot yourself in the foot, but what language doesn't?

Good lord.

Yes it is quite expensive. The issue is memory bandwidth is a lot more constrained when youre routing to hundreds or thousands of cores instead of the handful you need to support on a CPU.

GPT-3.5 is much, much smarter than Llama2. Its not nearly as close as the benchmarks make it seem.

Isn't this just ignoring all the lessons we learned from pretraining large models? Seems like a different flavor of the bitter lesson.

Yes kafka is definitely in an awkward latency spot.

I'd guess these model's understand works more closely to people so encoding in text is more token efficient and things like comments help.

Also syntax seems a lot easier to understand for them than semantics/logic. If you've used GPT-4 it almost never makes syntax errors. Logical errors on the other hand...

Since the cost per token of a forward/backward pass is so high on the models, the relative cost of a training set is small, and bandwidth (not talking about interconnect bandwidth) is even cheaper relatively.

Multiple servers adds a pretty hefty layer of networking, orchestration, security, failure handling (even in the datacenter), and serialization - even if the business logic stays mostly the same.

I'd be surprised if 99.9% of the targeted userbase of this app cares about this. I don't.

This is absolutely not true, having shared memory or even just being able to communicate over local os pipes is massivley simpler than introducing a network.

I mean you'd get the same number of page faults with both examples, so I think the discrepency is explainable by caches/cpu prefetching.

This is one of the most common arguments for non reference counted GCs

Lockless is not the same as lock-free, which is not the same as wait-free which seems to be what you are describing

It would be cool to see this compared with a more simple bin packing technique, or even just stuffing the most commonly used instructions at the top and those ordered by size.

Atomics scale very well if you are reading often and writing rarely.

Good article, but if i had to guess the subsequent L3 cache access after an increment is likely far overshadowed by the overhead of coherence messages, or the cores communicating between each other.

Doom is the same. It uses evil mode by default and SPC as the default leader key.

Also in my experience it is a lot faster.

It's hard to blame interest rates for all the wealth in tech when you look at all the new products and innovations that have happened over the last 10 years.

Huh? pretty much every one of the recently successful AI startup employees (OpenAI, Anthropic, etc.) have had stints at Google Brain.

There might not be a "Z" token for some of these names. A made-up example is "Lukasz" might tokenize to ["Luk", "asz"], so the model doesn't have any notion of how words are actually spelled. I suspect that if the body of training data came with some instructions on spelling it would know how to do this better, but it seems unlikely that there would be a natural language description of how Polish (?) names are spelled in the training data.

Could I get a source for that? Not that I don't believe you, but my napkin math puts the cost of training the 65b parameter model alone at a lot higher than 100k.

I see a bullet point saying Varies is for "random or non-constant state"

> Trained using the Chinchilla formula, these models provide the highest accuracy for a given compute budget.

I'm confused as to why 111 million parameter models are trained with the Chinchilla formula. Why not scale up the training data? If you're training smaller models, surely optimizing performance is better than optimizing total compute.

Seems like a silly misunderstanding of the Chinchilla paper, but I'm sure I'm missing something

Yes! I have been spending the last couple months pulling out completely unnecessary redis caching from some of our internal web servers.

The only loss here is network latency which negligible when you're colocated in AWS.

Postgres's caches end up pulling a lot more weight too when you're not only hitting the db on a cache miss from the web server.

Would an ASIC really be better than GPUs? I'm not 100% sure but aren't low power GPUs essentially ASICs for matrix multiplication.

Maybe the output was long enough that the context window would miss your previous prompts?