skohan

Be the change you want to see in the world

I think the 16" pro is already at the max battery size you can take on a plane.

Loading data to the GPU will generally never be the bottleneck. Yes that's a bit better for unified memory, but prefill and decode will still be bound by compute and memory bandwidth respectively.

There's some value in having a huge amount of memory addressable by the GPU for loading larger models, but even that doesn't make sense for current gen models. For MOE models, newer inference engines with smarter GPU expert caching and pinned memory allow a GPU with lots of bandwidth to significantly outperform a Spark, so that leaves you with larger dense models that fit on a spark but not on a GPU where you might have a use-case.

But even then it's hard. For instance a full-fat BF16 of a ~30B parameter model will fit on a 128GB spark and not a 32GB 5090, but at that size it's going to be frustratingly slow on the spark given the compute and bandwidth constraints.

Imo these devices are mostly good for experimentation, or if you only want to do overnight jobs or training which wouldn't be possible on a GPU, and you don't want to pay $100k for a DGX station.

Base M7 probably will still have pretty low memory bandwidth for serious AI workflows. M7 max/ultra might be pretty interesting though.

Linux is pretty great these days

Memory size is a bottleneck, but I have found that speed is very important for getting actual work done. Running a big model is great, but if you're getting 10t/s you're going to be spending a lot of time waiting.

And the cost of switching is near zero. It's not like trying to get SAP out of your org.

I'm also local-only in my current development process, and have a couple local agents running right now. The problem with local right now is the price has gone through the roof. Even 3-6 months ago I would have said it's viable for a lot of people, but even getting the RAM for a local inference machine is a big investment for a lot of people trying to dip their toes into the space these days.

The network effect is way weaker for LLM's. Switching from AWS to GCP or Azure is at least a multi-month project for an engineering org, if not completely operationally impossible. Switching LLM providers is at worst asking your engineers to install a new coding agent client, and at best switching a URL and API token in your config file.

Yeah it's a bit weird that it gets a different accent for part of the video.

But some people are not great speakers - I can imagine using AI narration if you don't feel like your own speech will meet your own standards.

> My tech career was just converting specs from clients and employers into machine code, with extra steps.

I guess the thing about that process is, you inevitably complete an incomplete spec. There are always fine details that will never be decided until they're required to complete the implementation.

I mean the labs are in a tricky spot where they have to justify massive valuations. "We're going to stay where we are and focus on inference" isn't the growth story you need when you're trying to justify a 2T valuation on 80B in revenue (unless you dress it up as a safety concern).

Google can afford to take their foot off the gas without killing their business.

No reason you couldn't in principle

You misunderstood my argument.

The point is, there are some domains (data-oriented programming) where Rust actively resists the kind of work you need to do to solve the problem in an optimal way, and it makes sense to use a language built around the type of memory management required.

There's no point in using Rust when 90% of your work is deactivating the benefits Rust provides and working around its default behavior.

It feels wasteful but it works.

I have ~15 years experience with hands on engineering. I also served as a hiring manager in a couple of roles, and interviewed dozens of candidates.

I would love to live in a world where every candidate represented their experience accurately, and every applicant was capable of fulfilling the role, but sadly it's not the case.

I agree there's some filters which are mostly performative (like leetcode), but being able to talk through the kind of work you do is table stakes. And not just for engineers, for any professional role.

I have interviewed dozens of engineering candidates. A big part of what I am looking for in an interview is how the candidate thinks, and what it's like to work with them in an interview.

If someone can't communicate well on technical matters, that's generally a negative.

That said, if a candidate was frank with me that they don't do well being put on the spot with a question like that, and can articulate how they do actually solve problems, and give examples of how it's worked for them in professional settings, I would consider them for the role if their answer was convincing.

I.e. if I asked you to explain how you would approach a problem, and you said you might need to go off and write some throwaway code to explore the edges of the problem that's a good answer.

In principle a large enough neural network can provably approximate any continuous function. Since arithmetic can be represented with a concrete grammar, it's not surprising to me that an LLM could contain an abstract representation if trained correctly.

With all due respect, are you sure you're not the one being dogmatic here?

90% of the code I write these days is Rust. It's a great tool, and the performance/safety balance, as well as general developer experience makes it well-suited to a wide variety of applications.

But in the past I've also built an ECS implementation. For that to optimize performance (and yes I did measure this) you want to do bulk memory allocations memory, pack bytes in order to minimize cache misses, and dispatch multiple threads to read and write concurrently over your memory pool. You can do all that in Rust, and people have, but it's a pain in the ass.

And for example, if I were working on a high-frequency trading application, where every millisecond of additional latency represents millions of dollars of lost revenue, I wouldn't waste my time fighting the burrow checker just so I can have a prettier program.

Rust is an amazing tool, but that doesn't mean it's the optimal tool for every job.

I think it's hard to understand how much better your experience could be if all your software was 10x faster and using 0.1x the resources, since you can't test both experiences side by side

It's like riding a bike with a tiny bit of resistance in the bearings. You will get used to it, and will probably feel like your bike is fine. But if you replace the hub and remove that friction, you will wonder how you ever tolerated riding a bike that way.

The logic here is a bit different. Consumer software is slow because it's the user paying for the compute to run those bloated Electron apps. Unless we're all running local inference in the future, it would be the vendor paying for the compute to run the AI-based UI.

Of course they would pass that along to the consumer, but nobody is going to pay token-based pricing for something that used to be free and ad-supported, or a cheap monthly subscription.

It might be fine for one-off applications, but it's incredibly inefficient for something that has to be done over and over.

Why would you use a GPU and gigabytes of VRAM running a non-deterministic process for something a program can do on the CPU in milliseconds with kilobytes of memory?

The Linux kernel has to balance security and performance concerns, security probably being primary, so it makes sense to make the tradeoff for more safety with worse ergonomics for memory management.

If all you care about is performance, you're going to be fighting Rust's memory model for no reason.

Idk about Qwen 2.5 Coder that this is based on, but modern models don't struggle with arithmetic. I see Qwen 3.8 do long-hand math in thinking traces (ie. it outputs 17+35+12=64 in the thinking trace) so I would imagine this is something they train into the model specifically. Sometimes it creates and runs a simple python script to check its work.

We had decades to work on making software more performant, but instead most users' space-age 2nm chips spend most of their time running Javascript because it was faster to make software that way.

I mean you can, but C/C++ is way more ergonomic for dealing with memory layouts.

The US is medium trust.

Open router doesn't host the models, so they wouldn't have access to the weights or reasoning traces right?

Infra might very well be the right play. Over the past year, it seems like the evidence is on the side that there's no "secret sauce" in AI, and models will all converge towards similar capabilities. Not much margin to be had for having the best model for a month or less until another lab leapfrogs you by 3%.

C/C++ is still relevant for HPC, where you want to carefully optimize memory management.