5,755 karma · joined September 4, 2014
Persistent weight adaptation also happens just not in real time - sessions are captured, analyzed, transformed into training data, fed into SFT/RL environments and later contribute to model updates. Takes a bit of time for the whole loop but you can't say it's not present.
There's nothing fundamentally preventing real-time weight updates, ie. LoRA-style online adaptation would be one obvious approach. It's just generally not worth doing at scale. Updating a shared model centrally gives much better data efficiency, batching, evaluation, control etc. than continuously training a separate set of weights for every user/session.
There is some work happening on narrowing that gap, for example Mistral has been pushing efficient LoRA-based customization, continuous pretraining, model adaptation etc.
I also did play a bit with activation steering – it's super cool where you extract profile for some concepts (emotional in my case) and you have effectively toggles to control "brightness/contrast" those areas (enhancing or suppressing those activation regions from profile) injecting to the model those concepts (emotions in my case) – you can do it in real time and it's fun thing to play with.
When somebody says that colleague is "really good", it's a good judgement signal for me.
My personal guess is that it's one of those. With effective context engineering it's hard to use all 20x quota, the limit becomes your own attention and time really.
You may argue that you're doing multiple, parallel extreme effort tasks – which may be true but then again, there will be results to actually look at sooner or later and that takes time.
Single user conversation spawns multiple parallel backend conversations, you need extra room for it as well, not just single context.
This plus usual apps like Mail, Spotify, iTerm2, SourceTree etc. also fill in memory.
Also 4 bit quantization is already quite aggressive compromise (measurable but sometimes acceptable loss, compared to ie. 8 bits which are often practically lossless) – for weights it's ok'ish, sometimes (especially if model was trained as 4 bit quants aware), but activations need to stay at higher bits taking more memory, otherwise quality degrades a lot.
For a dedicated headless setup, I’d probably use something like NVIDIA DGX Spark rather than a Mac (to be more precise NVIDIA GB10 Grace Blackwell from other suppliers than directly NVidia, they are much cheaper and have same insides). Linux is much better for running headless server, you also get standard NVIDIA/CUDA ecosystem instead of being tied to Metal/macOS.
For people who are interested in buying IMHO I'd wait a bit – next generation of Spark and/or Macs that are going to come out next year will be much better / will cross the line of being actually useful, not just a toy with goldfish LLM.
64GB+ or dedicated 48GB (2x24 on GPUs) is IMHO absolute minimum.
It lost 69.2% -> 67.8% on MMLU while improving medical performance. If you're trying to argue that loss of ~1.4 points is "catastrophic forgetting" (it's not) then look at later work, ie. Me-LLaMA that clearly demonstrates continued pretraining that improved both general MMLU and medical performance.
Not sure why you're fixating on catastrophic forgetting. How do you think model training works? Model training is just a sequence of checkpoints: pretraining produces it, training resumes from last, continued pretraining starts from last, supervised fine tuning starts from last, RL/post-training starts from last - it's just a sequence of checkpoints.
There isn't some fundamental distinction where original author continuing training from checkpoint X is training but a third party downloading checkpoint X and continuing training from it suddenly isn't.
ie. checkpoint doesn't somehow become a different kind of artifact when it's published.
Model weight release is a snapshot/checkpoint you can take and resume training on new data, producing new model.
You don't need original training history to modify it further.
Weights aren't just executable artifact that's consumed by users. Third parties actually use released parameter state as the editable starting point for further training and produce new foundation models from it.
Models are lossy compressed datasets you can pick up and amend (fine tune / continue training / alter) according to license they were released under.
Hy4 is released under OSI approved Apache License 2.0.
I settled on simplicity, extended it to serve clear, useful purpose.
Started with markdown database, single fact per line, structured/parseable (subject VERB object). This can be easily diffed/reviewed etc in git. Then added optional metadata (confidence, tags, persisted comments for natural language, uncertainty for numeric values, context, timestamps/spans), querying, alternative sqlite3 backend, self describing VERBS, z3 solver etc.
Those kind of graph information systems are great to quickly structure knowledge in a way that LLMs and humans can use/act on/loop on. Creating ontology, linking, some rules and actions and kicking it so it all munches and spits out results that feed back in so it self evolves. It's very natural for llms to query/update/restructure those graphs (also good for humans because it's very terse, essential information only).
There is no need to create k8s style complexity/services/what-not, it all works well from single sqlite db file or bunch of markdown (.cave in my case) files.
It's also interesting to see how well local open weight models are dealing with information arranged this way.
they just don't recommend it.
nodejs stopped bundling corepack since node 25, maybe you're referring to that?