lqstuart

Man you’re really good at that lol

I’m saying actual algorithmic (as in not data) model innovation has never been a significant part of the revenue generation in the field. You get your random forest, or ResNet, or BERT, or MaskRCNN, or GPT-2-with-One-Weird-Trick, and then you spend four hours trying to figure out how to preprocess your data.

On the flipside, far from figuring out GPU efficiency, most people with huge jobs are network bottlenecked. And that’s where the problem arises: solutions for collective comms optimization tend to explode in complexity because, among other reasons, you now have to package entire orchestrators in your library somehow, which may fight with the orchestrators that actually launch the job.

Doing my best to keep it concise, but Hopper is like a good case study. I want to use Megatron! Suddenly you need FP8, which means the CXX11 ABI, which means recompiling Torch along with all those nifty toys like flash attention, flashinfer, vllm, whatever. Ray, jsonschema, Kafka and a dozen other things also need to match the same glibc and glibc++ versions. So using that as an example, suddenly my company needs C++ CICD pipelines, dependency management etc when we didn’t before. And I just spent three commas on these GPUs. And most likely, I haven’t made a dime on my LLMs, or autonomous vehicles, or weird cyborg slavebots.

So what all that boils down to is just that there’s a ton of inertia against moving to something new and better. And in this field in particular, it’s a very ugly, half-assed, messy inertia. It’s one thing to replace well-designed, well-maintained Java infra with Golang or something, but it’s quite another to try to replace some pile of shit deep learning library that your customers had to build a pile of shit on top of just to make it work, and all the while fifty college kids are working 16 hours a day to add even more in the next dev release, which will of course be wholly backwards and forwards incompatible.

But I really hope I’m wrong :)

Comments like this view the ML ecosystem in a vacuum. New ML models are almost never written—all LLMs for example are basically GPT-2 with extremely marginal differences—and the algorithms themselves are the least of the problem in the field. The 30% improvements you get from kernels and compiler tricks are absolutely peanuts compared to the 500%+ improvements you get from upgrading hardware, adding load balancing and routing, KV and prefix caching, optimized collective ops etc. On top of that, the difficulty even just migrating Torch to the C++11 ABI to access fp8 optimizations is nigh insurmountable in large companies.

I say the ship sailed in 2012 because that was around when it was decided to build Tensorflow around legacy data infrastructure at Google rather than developing something new, and the rest of the industry was hamstrung by that decision (along with the baffling declarative syntax of Tensorflow, and the requirement to use Blaze to build it precluding meaningful development outside of Google).

The industry was so desperate to get away from it that they collectively decided that downloading a single giant library with every model definition under the sun baked into it was the de facto solution to loading Torch models for serving, and today I would bet you that easily 90% of deep learning models in production revolve around either TensorRT, or a model being plucked from Huggingface’s giant library.

The decision to halfass machine learning was made a long time ago. A tool like Mojo might work at a place like Apple that works in a vacuum (and is lightyears behind the curve in ML as a result), but it just doesn’t work on Earth.

If there’s anyone that can do it, it’s Lattner, but I don’t think it can be done, because there’s no appetite for it nor is the talent out there. It’s enough of a struggle to get big boy ML engineers at Mag 7 companies to even use Python instead of letting Copilot write them a 500 line bash script. The quality of slop in libraries like sglang and verl is a testament to the futility of trying to reintroduce high quality software back into deep learning.

I like Chris Lattner but the ship sailed for a deep learning DSL in like 2012. Mojo is never going to be anything but a vanity project.

The fact that there’s even a debate about banning smart phones in classrooms tells you all you need to know. Cell phones were de facto banned in school in like 2002, not sure when it became the norm but it seems like a no brainer.

In fairness, the biggest breakthrough in AI has been calling more and more things “AI.” Before LLMs it was content based collaborative filtering.

It’s frequently suggested by people with no background and/or a huge financial stake in the field

“AGI” isn’t a thing and never will be. It fails even really basic scrutiny. The objective function of a human being is to keep its biological body alive and reproduce. There is no such similar objective on which a ML algorithm can be trained. It’s frankly a stupid idea propagated by people with no meaningful connection to the field and no idea what the fuck they’re talking about.

I tried it once and couldn’t understand why it exists. Neither windsurf nor cursor seem to offer anything Cline doesn’t, and the real Microsoft developed tools like Pylance are also broken and can’t be replaced with shitty AI.

Windsurf already had sold their code and most of their devs to DeepMind. The company is worthless, idk what Devin thinks they’re buying.

This is definitely true. You see it constantly in deep learning applications, where eg NVIDIA’s fancy fp8 stuff needs C++11 ABI, so they need to compile torch from source, which means everything with C++ dependencies on torch needs to be recompiled, and eventually it’s much easier to ship a container image. It can be done with an Ansible playbook, sure, if you don’t have to ever do anything else with your machine.

Haha yeah. I’ve seen you mention the llama cpp wrapper elsewhere, it sounds cool! I’ve worked enough with vLLM and sglang to get angry at xgrammar, which I believe has some common ancestry with the GGML stack (GBNF if I’m not mistaken, which I may be). The constrained decoding part is as simple as you’d expect, just applies a bitmask to the logprobs during the “logit processing” and continuing as normal.

I don’t really see a need for token IDs alone, but you absolutely need per-token logprob vectors if you’re trying to do constrained decoding

Counterexample: the Linux kernel

Stephen Hawking had a pretty good idea as to how and why - https://en.wikipedia.org/wiki/Hawking_radiation

This is not to be confused with the infrared light that’s relatively easy to detect, which isn’t leaving the black hole but is just light emitted from matter heating up due to tidal forces near the black hole.

I don’t think humanity is ever really going to know what a black hole actually is, simply because they’re too far away and the energy levels required for experiments are many many orders of magnitude beyond what we can generate on Earth. But I do expect if we could get up close and study one, we’d simply discover a lot of fascinating new physics over top of some kind of matter not that far off from neutron stars.

A lot of people don’t really get past the pop science articles about “singularities” to realize that those are just failures of our theories. We don’t know how anything behaves when gravity is strong enough to affect the quantum level, because QM has some kind of silly techniques for avoiding it under normal circumstances (called renormalization) that no longer work.

At extremely high energy levels, my guess as a guy on the internet is that the universe looks less like magical portals to Matthew McConaughey’s bookshelf, but more like regular boring space crap that destroys our notions of stuff like time, matter, energy, causality etc. And in fairness, neutron stars already kind of do this, but nobody cares for some reason.

TIL there’s a Nancy Grace telescope

This is really old

People started using search engines to ask stupid questions. An LLM like Gemini etc is hands-down better for that. A search engine is still better for actually searching. I do not need a 5000 word screed about a guacamole recipe.

So by “we hacked Gemini and leaked its source code” you really mean “we played with Gemini with the help of Google’s security team and didn’t leak anything”

I have no comment on any of the general whining about the ineffectiveness about academia and the public sector, but what struck me about this article is how tiny the funding amounts are. They’re talking about a £40 million endowment like it’s impressive? Maybe I’m just too American but it’s baffling that Cambridge or UCL cares about a £1 million grant.

This appears to be a bot or some kind of malicious actor that contributes nothing but political propaganda

I think this spirit is totally lost on most people in this field. It’s tempting to say younger generations but it’s everyone. It always amazes me when I meet someone who has spent 10+ years in this field and doesn’t even care how anything but their shitty Kafka-Flink pipelines work.

I'm an American, and I have worked for both Zuck-owned and Chinese-owned companies, and you really, REALLY might want to re-think your stance. It's extremely ironic that we as Americans have this "question authority" thing so deeply ingrained in our DNA that it pushes us towards authoritarianism just to be contrarian.

In all the internet posts about this, I've never seen a response to the fact that China has been doing this to US tech interests basically forever, and even the EU has done its utmost to severely handicap US tech companies for the same reasons. Why hold the US to a double standard against our own interests? TikTok is the exact same garbage you can get from Insta, YouTube, for fuck's sake even LinkedIn has video now. What is the incentive to allow a hostile foreign power that repeatedly and overtly attacks our infrastructure to control the biggest player?

> I’m very unexperienced when it comes to reverse engineering

Lol… where does that leave the rest of us in comparison

I used to be one of those kids that hated on Java for being boring, until I spent six years doing nothing but deep learning in Python and C++. I miss you, Gradle…

Articles like this are a nice reminder of just how wonderful life is in America, and just how far civilization has to fall if shit REALLY hits the fan someday. If you need a reason to be grateful in spite of mundane crap like housing costs and white collar jobs, the developing world offers some lessons in humility and gratitude.

…and then they sold out to a Wall Street dickhead, and here we are

That’s because you don’t understand what group A strep is or what it means to eliminate it

Nothing wrong with Elon making big bets if he wants to.

Polymarket is hardly an efficient market. They also show Republicans winning both the presidency and popular vote as about as likely as them winning just the presidency, despite virtually everyone giving Democrats 70/30 odds of winning the popular vote (as usual). https://polymarket.com/event/which-party-wins-presidency-pop...