That is literally the thing the parent poster wants to avoid by running open models.
[edit] I was a little unfair -- lack of access to training data is a bit of an issue (perhaps moreso for analysis than for for actual use, considering what it takes to train these models). I'm thankful that some of them are also distributed as base models, which should be relatively unbiased compared to what happens later during finetuning.
You'd have to be doing something where the unified memory is specifically necessary, and it's okay that it's slow. If all you want is to run large LLMs slowly, you can do that with split CPU/GPU inference using a normal desktop and a 3090, with the added benefit that a smaller model that fits in the 3090 is going to be blazing fast compared to the same model on the spark.
Yeah, by "structured" I mean how it wants to do ChatGPT-style output with headings and emoji and lists and stuff. And the punchy style of K2 0905 as shown in the fiction example in the linked article is what I really dislike. K2 Thinking's output in that example seems a lot more natural.
I'd be totally on board if cut straight to the answer with natural sounding prose, as you described, but for whatever reason that has not been my experience.
Hi Simon. I have a Xeon W5-3435X with a 768GB of DDR5 across 8 channels, iirc it's running at 5800MT/s. It also has 7x A4000s, water cooled to pack them into a desktop case. Very much a compromise build, and I wouldn't recommend Xeon sapphire rapids because the memory bandwidth you get in practice is less than half of what you'd calculate from the specs. If I did it again, I'd build an EPYC machine with 12 channels of DDR5 and put in a single rtx 6000 pro blackwell. That'd be a lot easier and probably a lot faster.
There's a really good thread on level1techs about running DeepSeek at home, and everything there more-or-less applies to Kimi K2.
Interesting, I have the opposite impression. I want to like it because it's the biggest model I can run at home, but its punchy style and insistence on heavily structured output scream "tryhard AI." I was really hoping that this model would deviate from what I was seeing in their previous release.
If you want to do it at home, ik_llama.cpp has some performance optimizations that make it semi-practical to run a model of this size on a server with lots of memory bandwidth and a GPU or two for offload. You can get 6-10 tok/s with modest hardware workstation hardware. Thinking chews up a lot of tokens though, so it will be a slog.
My employer was running Growthpower (ERP software) on an HP 3000 system up until 2018 or so. We replaced it with a "modern" .NET/MSSQL ERP solution that does a lot more, but it's slow and terrible to navigate compared to the old console menu system, and its database is hundreds of tables without a single foreign key. The frontend application makes a long series of sequential queries to build each view... if you're willing to wade through the muck, you can write a server side query that can do in milliseconds what it does in minutes.
I don’t own a laptop. I run DeepSeek-V3 IQ4_XS on a Xeon workstation with lots of RAM and a few RTX A4000s.
It’s not very fast, and I built it up slowly without knowing quite where I was headed. If I could do it over again, I’d go with a recent EPYC with 12 channels of DDR5 and pair it with a single RTX 6000 Pro Blackwell.
I used Omron's K3GN panel meters in a project at work and I had to draw the alphabet in the configuration drawing because it is so unintuitive. It's not a whole lot worse than the one shown in the article, but still... it's pretty rough. I think I prefer numbered parameters like you typically see on VFDs. It's a lot easier to just scroll to P148 or whatever, enter to view/modify, scroll the value, enter to set. Menu trees on seven-segment interfaces are a mistake.
Yeah, the whole "let's come up with a slur for <blank>" thing entices people to build their fictional racism on real racism, and it just devolves from there. I saw "wirebacks" thrown around recently, among others.
You might be able to do "large models slow" better than the spark with a 5090 and CPU offload, so long as you stick with MoE architectures. With the kv cache and shared parts of the model on GPU and all of the experts on CPU, it can work pretty well. I'm able to run ~400GB models at 10 tps with some A4000s and a bunch of RAM. That's on a Xeon W system with poor practical memory bandwidth (~190GB/s), you can do better with EPYC.
CPUs with stock cooling solutions will turbo boost up to max temp and stay there, that's completely normal and shouldn't cause a CPU to physically burn up, even if you do it for months.
Delaminating the PCB is absolutely wild. I'm really fascinated by the idea of doing meetings in Second Life, though. What did people use for avatars at work?
Alex Jones made his money by convincing people that the parents of children murdered at Sandy Hook Elementary were actors in a conspiracy, which resulted in them bein harassed endlessly. He even continued to attack them during the defamation trials. I think it's unfair to say he was merely "the closest vile person around with big pockets," he really was the cause of a lot of pain.
Great tip! Apparently that's readline's config file, so this will affect a lot of things. That's great news for me; after switching to zsh I got used to case-insensitive tab completion, and now it annoys me when other tools don't work that way. This should help a lot.