robrenaud

I was surprised in reading the article that it had nothing to do with the contents of the model. It could have been any model capable of 1M context that triggered this bug/misconfig on Cloudflare/SGLangs side. The title is extremely misleading.

Are there inference products focused on say, optimizing frequently used internal contexts, so that they are compact in the kv cache? Perhaps there rep could be optimized over hundreds+ of distinct real use cases?

Maybe your law firm doesn't want your data going outside your business. But you obviously want high quality analysis.

What is inefficient is thinking in 5000 floats, and then quantizing all that computation into a 17 bit token, just to immediately turn the token back into a vector.

Hallucinate -> do an experiment -> see it fails, try again. Hallucinate -> do an experiment -> it works, model innovated.

Agentic models brush up against reality, this gives a way around the hallucination problem.

Here is a recent talk showing that hallucination and discovery are actually positively coupled. https://www.youtube.com/live/ZNlZsI9kBm4?si=nhn4ancXu7s6qtom...

Brains aside, it S1/S2 seems like a good description?

System 1 is quick and intuitive. System 2 is slow and deliberate.

Compared to models that compose responses with many forward passes, thinking in token space, these models make a quick decision after processing their inputs.

Here is a law I wish we had for social media. Let's just align incentives.

If you are working on a social media platform, and your total comp is >= 500k, your children must use your platform for the average amount of time that children of that age range use your platform nationwide.

Look up the double descent phenomenon. Large models with lots of data generalize rather than overfit.

Also see the argument around figure 3 here. https://arxiv.org/pdf/2503.02113

Regarding the temperature explanation:

> "Instead of picking the highest-probability token, we can use different selection strategies to balance safety and creativity in the generated text".

Safety is definitely the wrong word here.

Temperature 0 generated text actually has a weird "lack of surprise" character that makes it seem artificial. [1]

> "high-probability texts can be dull or repetitive. Humans use language as a means of communicating information, aiming to do so in a simultaneously efficient and error-minimizing manner; in fact, psycholinguistics research suggests humans choose each word in a string with this subconscious goal in mind."

I'd completely drop the dropout explanation. It's just not part of the modern recipe anymore, AFAICT.

As for the ambitious goal of explaining transformers with a single interactive visualization, I just have a hard time imagining a person is going to newly understand both word embeddings (word2vec blew my mind in 2014) and also gain an understanding of attention.

I am making my own visualizations for a presentation on "Full Bandwidth Transformers"[2] that I am giving tomorrow at the Deep Learning Study Group (SF) (on zoom for the non-locals)[3]. It's not meant to be stand alone/context free, but I'd love some feedback.

https://rrenaud.github.io/fullbandwidth_transformer_viz/

[1] https://arxiv.org/abs/2202.00666 [2] https://arxiv.org/abs/2608.08888 [3] https://www.meetup.com/deep-learning-sf/events/316601593/

In the Jev use case, LLMs are horribly uncalibrated. In general, they will not produce good probability estimates.

Their generality also comes with a latency/computation costs.

> That's an interesting choice. One question I had when looking at the jev copy on their blog is if one "line" in their output looks / attends to other lines. I think not, since they say it's parallel and not autoregressive.

I don't understand the connection between the lack of autoregression and options attending to each other.

Non autoregressive models can attend to all the inputs simultanously.

An autogregressive model can can attend to all the options in the context of each other by simply writing the options out twice. Autoregressive models actually requires this, since one of them will come later, and the earlier prefill inputs can't attend to the later ones.

> “Predicting sales conversion probability from conversations using pure Reinforcement Learning”. Can you understand what that means?

I can understand it, and it wouldn't excite me at all.

Jev has a beautiful API and is advertised as something much more general.

> Using briefer abstractions in place of nested loops is almost always preferable.

Indeed, this is why everyone knows the J programming language.

Neal Nanda's walkthrough of it is pretty good.

https://youtu.be/KV5gbOmHbjU?is=GSiv0rSHBocNdU8l

Also long, but it's detailed and complicated, so there is no escaping that.

The part where he chooses his inputs to hit worst case behavior in Python's hash function.

Experts vary per token in MoE, there is maximum flexibility. Good for driving down loss, bad for locality/gpu memory/bandwidth.

If expert selection were more constrained, inference systems could take advantage of it. Keeping experts cached would mean not needing to load them from disk/ram every token.

Should LLMs be designed to be modular, so that instead of needing access to the whole model, for a given prompt, only a small subset of the model would be used? If knolwedge was sufficiently modularized, most of it could be ignored.

Maybe a hyopthetical model of 5T of indexable weights could be used with only 50 GB of GPU ram, efficiently, because it stays resident in the GPU.

It's cool that it proves that a bunch of vectorized outputs from an unknown embedder on an unknown dataset is in no way private, because of this ability to reverse engineer the embedder.

I talked to the author at his poster session at neurips and was able to get the gist, though I had read a lot about the platonic representation hypothesis, and this was one of my top 10 favorite papers in the conference.

I think a less stringent, but still workable assumption is that for very similair objects, their distances will be small. This is much easier to accomplish than agreement across all pairs.

The number of steps between token outputs is increasing.

I don't actually think this is a problem, but it is a further step towards inscrutability.

If black and white had different permutations of the back row, you could find more balanced starting positions. Adopt more flexible starting conditions in the computer chess age.

Does the model quality become degraded in other ways?

Do consider the incentives of those developing the model that made those predictions. Afaict, it was not selected for purpose other than testimony.

How do you get personalization without surveillance?

I suspect idiomatic TypeScript or idiomatic F# are both way better solutions in the real world than abstruse Typescript emulating idiomatic F#.

If a good SWE is $150/hour, does the model cost actually matter? Surely you'd be willing to spend $10/hour to make that SWE 20% more productive? The model cost is still much less than the salary.

They use a lightweight adapter to silently degrade the performance. Usually these adaptors are made to improve the performance for a given domain/task.

3 blue 1 brown has a great visual introduction to transformers, the heart of LLMs.

It's chapter 5. Start at chapter 1 if you want more background on neural nets and backprop.

https://youtu.be/wjZofJX0v4M?si=HFXbrB-5cArprGaU

"The reasoning is the weights."

The reasoning is in a process that uses the weights.

Sorting algorithms are just bytes. Those bytes don't sort by themselves. They do instruct a computer on how to sort though.

There is some recent work on modularizing knowledge in LLMs.

https://arxiv.org/html/2605.06663v1

It might be possible to train a big generalist that is a composition of modules, some of which can be dropped dynamically at inference time, depending on the prompt.

Is every American tax payer morally compromised?