It could be interesting to experiment with more complex representations in the memory/file system, i.e. vectors. Seems like the full advantage of this architecture is yet to be explored. Blending neurons with deterministic compute to fit on one GPU is a fascination concept, never considered it before!
First class hardware support makes this really exciting and different. On device processing is cheaper and faster, critical for real time applications of AI. Really excited for the rise of local, smart intelligence!
Whilst general self-reporting of confidence will be hard to achieve (we can't even do it as humans), I'm sceptical of the notion that we can't train models to give some confidence scores accurately. I think over the past year that with the correct RL, models can showcase some pretty insane capabilities. If reporting the correct confidence was a RL training target say for predicting weather events or a game outcome that is supported by real RL environment or simulated data, then I suspect LLMs will develop a strong intuition for confidence similar to how the recent JEV and decision models are trained.
Awesome write up and intro by a fellow Sydney-sider, kept me hooked! As someone who's not a maths wizard, the technical details flew by my head but the prose allowing me to empathise deeply with those in the field was very enlightening. I never realised how big of a deal the maths dump was and how much it has shaken up the field. I guess we are all just trying to figure out this whole "intelligence on tap" thing together.
I often think of this question. Probably things that are enduring. How to be a good person, how do people work, how does society work, how to learn, some hobbies and life skills.
I've been building voice agents for the past year, and honestly the word error rate and how good they sound are "good enough" across countless providers. The real gaps are in latency, handling interruptions, interrupting the speaker if needed (this is surprisingly useful in most contexts) and adapting to the emotion/tone of the context. Would love to see a benchmark or tests that measure these effectively and test the claims of Voxtral 3! These existing benchmark results look very impressive.
in terms of raw token throughput and serving users it is, but in terms of R&D and long term lab capabilities, training is still the goose that lays the golden egg
Supporting Huawei Ascend 950 NPUs with open source inference code has huge implications in the global AI hardware market. Imagine if all smaller labs now can build on top of this stack and add increasing support for more compatible hardware , software and models. This could seriously challenge NVidias death grip. Although inference is still only a part of all AI workloads and open sourcing training on Huawei chips would be even bigger.
People love convenience. Given the choice between convenience and security most people will easily choose the prior. Security is also just hard. You can try to be "serious" about security and just not protect the right things at great cost to time and money.
For me personally, iterating is huge. You can't arrive at a hugely complex system in one shot. I think of the LLM as someone I'm running software engineering "experiments" with. As requirements become clearer, I discuss with it more and more and over time I'll gain a stronger understanding of the problem and possible solutions.
You'll also discover over time which systems the AI can handle fine on its own and which abstractions and systems it just has no ideas on how to design. Often I find I have no idea how to design those too. It's then useful to try different designs, test tradeoffs and refactor aggressively if needed.
Agreed, OpenRouter is competing in an increasingly crowded market and even a lot of cloud providers like Neon, Vercel and Cloudflare are rolling out their own gateways to vertically integrate with their hosting. It's hard to say what the true net usage of LLM look like in the world!
ChatGPT became massive because of it's generalisability without needing training a new model for every domain and for its zero-shot learning inference/learning capabilities. If this model can be scaled or easily fine-tuned, it can have huge implications on forecasting in domains where custom models are too hard or too expensive! I am somewhat sceptical and wonder if the big quant firms already have versions of this internally or if not, why no one has achieved these kinds of results until now.
Your feedback is valuable and correct, I'll extract the library into /core in the repo and also manually verify all the citations. I'll read into the prompt injection literature more deeply and turn this from a shower thought project into something more mature
Yes, to be completely honest this is a vibe coded project and I'm by no means a security expert. This was more of a fun, side project/experiment based on a shower thought. I admit it's not good/disingenuous to imply security knowledge, but for what it's worth, I just prompted Claude to research the latest papers on prompt injection and it made the claims on its own. Again this should not be an excuse for not reviewing the AI's output more carefully, so in the future I'll be more careful with LLM output and also present it as a vibe-coded project. Apologies, I'm just a noob in prompt injection security who doesn't know what he's doing :(
MDX is not secure by default, you'd be executing arbitrary JS code potentially sent by untrustworthy sources, which is advised against in the MDX website.