1,439 karma · joined October 26, 2024
You are also rolling the training of your models by size and the lat est small model can sometimes beat an older model.
For a while we thought you just train the big model first then distill smaller models from it but since we've learnt the importance of on policy RL (basically the way a small model figures out a problem is very diff from a big model so it's best to RL optimize each model distinctly)
Reactive optimizers could actually be helpful if you used them to dynamically tweak training data mix and use them to make choices about RL environments (if you recall from the Mimo training dashboards, it’s quite tricky to figure out how to value and chose RL traces during post training)
EDIT: Brutal! Apparently it doesn’t support VRR or 120 hz at 4K. Surprising miss to not support HDMI 2.1 properly.
Further EDIT: Basically we are entering 2027 with no way to get a tv add on box that can stream 4K apps (Disney+ etc. need Widevine L1), 4K at 120hz (i.e. support HDMI 2.1 properly) and decode AV1. Just a bizarre gap in the market.
It doesn’t need to be perfect on edge cases just translate back “reasonable” models.
Should be something that is fairly easy to eval etc.
I never thought I’d prefer WSL to even my MacBook for working with remote servers and dev but somehow I do.
I of course know the many ways to roll something like this for myself, yes dev containers are better for many things etc. but it’s wierd how good the ergonomics of a WSL like container are.
I've been jamming on a sort of Corewars (remember?) / Starcraft hybrid battler where LLMs write sandboxed Lua programs to control bots fighting out 10-100 vs 10-100 tank battles. Every quarter the LLM gets a full view of the situation and can reprogram all the bots to better adapt strategy etc.
It's good fun to watch - excited to share soon.
I’m curious about how to think about the dynamics of a LVT:
- I can imagine urbanism creating a flocking behavior that ruins neighborhoods in 5-10 year cycles. Coffee shop draws more commence drawing more affluent people pushing up LVT driving out residents faster than even the spectre of “gentrification”, only to collapse when a nearby area is cheaper for a cool coffee shop to start. (The problem specifically is that you’ve both driven out people and caused an inefficient overbuild in each area)
- Similarly how do you think about new uses for land emerging? If I lease desert land for a data center because it’s so perfect for it, instead of buying it … how (and when) does it show up in LTV increases? What if I trade you some other benefit to keep it out of LVT impacting records?
- Are you just costing the world coordination surplus by forcing high value enterprises to distribute themselves (inefficiently) just far enough apart that they don’t drive up each other’s LVT? That’s a deadweight loss.
I’m a fan of the idea - these are just some of the tricky challenges I don’t have a good answer to yet.
I’d love for a class in this future language to come packaged with tests, invariants, fuzzer parameters, profile targets / performance budgets with realistic inputs (on this 100 element array this should take no more than X clock cycles), race condition stress tests etc.
The compiler (or even linter) should optionally run some / all of these checks and succinctly report back (with knobs so the LLM can manage wall clock time).
Adding each of these should not be follow on steps.
Beyond this, debug hooks should be trivial to set (in code itself), so the LLM can trivially say show me the stack after the 9th time this function is called on this input to the program.
Did you even read it through carefully once? If you couldn't bother to craft it, why should I read?