valine

With a mixture of experts model you only need to read a subset of the weights from memory to compute the output of each layer. The hidden dimensions are usually smaller as well so that reduces the size of the tensors you write to memory.

What would it cost to get 512GB of VRAM on an Nvidia card? That’s the real comparison.

Probably helps that models like deepseek are mixture of expert. Having all weights in VRAM means you don’t have to unlod/reload. Memory bandwidth usage should be limited to the 37B active parameters.

There is no alternative. Europe tried to peacefully coexist with Germany in the 1930s. Standing up to imperialist dictators is the least bloody option.

My impression is that it’s a massive increase in the parameter count. This is likely the spiritual successor to GPT4 and would have been called GPT5 if not for the lackluster performance. The speculation is that there simply isn’t enough data on the internet to support yet another 10x jump in parameters.

O1-mini is a distill of O1. This definitely isn’t the same thing.

Lets just call every turning complete system AI and be done with it.

Optimization can be a tool used in the creation of AI. I'm taking issue with people who say their optimizer is an AI. We don't need to personify every technology that can be used to automate complex tasks. All that does is further dilute an already overloaded term.

My point is that it’s equally ridiculous to call either AI. If our chip here is not the AI then the AI has to be the optimizer. By extension that means AdamW is more of an AI than ChatGPT.

I strongly dislike when people say AI when they actually mean optimizer. Calling the product of an optimizer “AI” is more defensible, you optimized an MLP and now it writes poetry. Fine. Is the chip itself the AI here? That’s the product of the optimizer. Or is it the 200 lines of code that defines a reward and iterates the traces?

The RL is done on problems with verifiable answers. I’m not sure how o1 slop would be at all useful in that respect.

The existence of R1-zero is evidence against any sort of theft of OpenAI's internal COT data. The model sometimes outputs illegible text that's useful only to R1. You can't do distillation without a shared vocabulary. The only way R1 could exist is if they trained it with RL.

Results of their clinical trial look positive.

https://pubmed.ncbi.nlm.nih.gov/39289373/

Even if everything you said is true it doesn’t mean general AI is 20 years away. Computers are many orders of magnitude faster at non-parallel tasks. Who’s to say raw clock speeds won’t easily compensate for any deficiencies vs the brain.

You can run a trillion parameter model on a single machine today. We’re maybe 4 or 5 years away from that same 1T model running on an iPhone.

And human reasoning is somehow more magical? Really struggling to understand the distinction between searching through a latent space and “reasoning”.

You probably had your render loop set to poll rather than wait for events. If you don't configure wait events you'd be re-rendering your UI as fast as your monitor refreshes.

How did you create the rotated labels? That's been a major pain point for me using ImGui, I wish there was native support for that.

You can induce an activation potential with an external EM field. That’s the entire basis for deep brain stimulation.

No, and that’s because we are talking about relative positions. Every query can have its own set of keys. From the perspective of token 100 token 3 would be squished down, but from the perspective of token 3 it is still at position 3 and can see tokens 0,1,2 without them being squished.

They don’t need rapid reuse for Artemis. It would certainly help but Starship can just hang out in orbit for a few weeks while they do what they need to do to launch all their rockets.

One strategy I’ve been playing around with is to take an instruction I want the model to follow and squish the positional encodings for the keys down to position zero, and the new queries out slightly further in the window. The model will still follow the instruction but the behaviors are more global. It’s behaves more like a fine-tune and less like the instruction is part of the conversation.

You have to be careful to keep the relative positions for adjacent and nearby tokens intact. The relative positions of distant tokens are less brittle.

Maybe this is wishful thinking but could something like this be used to make star wars style holograms? Strategically placed laser shadows to block light sounds like a useful tool in that direction.

Some people are really great at reading code, and they’re the best equipped to take advantage of LLMs in their current slightly unreliable state. If you can read and fix the LLMs code faster than you can write it from scratch then it’s a net win.

One of the things I really love about rope is that it allows for a lot of interesting encoding schemes at inference time without model retraining. I’ve had a lot of fun playing with different relative positions. You can elicit a lot of interesting behaviors from the model when you use different rotations for keys vs queries, they don’t always have to match.

For example exact position doesn’t matter too much when tokens are spaced out. Let’s say you use token position 100 for your query, you can shift all the keys around position 100, and the further they are back in the context the more freedom you have to play with the value.

LEO is the exact place you need to be careful. Higher orbits have more space and have fewer satellites overall so it's less of a concern.

Obviously not a problem for IFT6 since it's sub-orbital, but the original comment was about why we need a deorbit burn rather than just triggering the flight abort system.

Detonating something in orbit could trigger Kessler's Syndrome.

Unless I’m reading it wrong I don’t think rows matter. Attention doesn’t take into account sequence position natively, that’s why positional encodings exist.

I would like to see a comparison for the inference time compute between a regular transformer and this. I’m assuming token/s is lower since you need to compute the weights of the model for each token prior to the actual attention calculations for the sequence position.

Completely agree. It’s frustrating how every thread on self-driving devolves into the same argument about cameras vs. LIDAR. Everyone points to Tesla FSD disengagements as evidence, when the vast majority of disengagements are caused by the route planning, not the vision stack.

I’ve watched countless hours of FSD footage and have FSD on my Model 3, and I’ve never seen it incorrectly map the position of a vehicle. Both Tesla and Waymo seem to have reliable perception stacks, the actual decision-making is the hard part, like you said.

People get hung up on the vision stack and mapping. Obviously, any self-driving car needs a way to perceive the world around it, and Tesla’s approach is very different from Waymo.

The software that actually does the driving, ie turns the wheel and works the pedals, is very similar between the two companies. Tesla took a big step in Waymo’s direction earlier this year when they replaced their optimization based path planner with a neural network.