My advice: don't just look at tokens per second, but also at time to first token (TTFT).
The local inference space is leaning to MoE models, and a lot of them have decent tokens / second, but horrible TTFT.
4,098 karma · joined January 9, 2014
The local inference space is leaning to MoE models, and a lot of them have decent tokens / second, but horrible TTFT.
A single M3 maxed can run a Q2 Kimi 2.6, though thats with a hardly degraded perplexity.
2x M3s with RDMA can run a lossless Kimi2.6 at Q4, but with CPU only you would get okayish decode but horrible (+1m) TTFT, that wouldnt be a great _interactive_ experience.
I have tried quite a bunch of local models, and the reality is that it's not just a matter of of "it's a small model that should be hostable easily". Its also a matter of whats your acceptable prefill TTFT and decode t/s.
All the local models I used, on a _consumer grade_ server (32GB DDR5, AMD Ryzen) have been mostly unusable interactively (no use as coding agent decently possible), and even for things like classification, context size is immediatly an issue.
I say that with 6m experience running various local models for classifying and summarizing my RSS feeds. Just offline summarizing ans tagging HN articles published on the front page barely make the queue sustainable and not growing continuously.
Well guess what, if you have a CRUD website and 100 users you're just not the target. Move on.
Some days ago I wanted to sketch a 3D model of my TV remote. I opened blender and what a mess of complicated windows and panes. I closed it immediatly. Do I think Blender is an over complicated mess? No, I just think I'm not the target. And I'm not offended to be too noob to use it.
I think what we need is the equivalent of what was done for CORS: client/server cooperation.
That is, APIs should mark that they are human only, and harnesses should cooperate with such flags and prevent calling said APIs.
It's not perfect, as it's client side enforcement, and one could still theorically build their own harnesses without, but that's the only way forward.
Plenty of other democracies have parties, including cross government branches.
What makes the US unique, and fragile, is that no party other than democrats and republicans can realistically exist.
It over emphasizes partisanship above anything (including honesty, morality) because career politicians in one party just have nowhere to go if they are dissident.
You can see that in plain sight currently, with republicans being in the total incapacity of contradicting their party line on anything, even the most obvious of lies.
In most other democracies, dissidents would have just created a new party and moved on, that wouldnt be "carrier ending" for them.
That is a single one liner of btrfs subvolume snapshot, in a single hook configuration file, ready to be valued at $10B as quantum agentic versioned sandbox startup.
No worries, there will be a startup creating "AGI Bench", >=80% means you're AGI, they will be valued $50B.
Would you? I am not very knowledgable on LLMs, but my understanding was that each query was essentially a stateless inference with previous input/output as context. In such a case, a single puzzle, yielding hundreds of queries, is essentially hundreds of paths dependent but individual tests?
SLAAC was part of IPv6 since the original RFC, its a horribly over engineered stateless replacement of DHCP. Nobody asked for that.
From my experience, this happens essentially by three means:
- Level 0 (beginner users) long lived conversations: If you dont get in the habit of compressing, or otherwise manually forcing the model to summarize/checkpoint its work, you will often find people perpetually reusing the same conversation. This is especially true for _beginners_, which did not spend time curating their _base_ agent knowledge. They end up with a single meta conversation with huge context where they feel the agent is "educated", and feel like any new conversation with the agent is a loss of time because they have to re-educate it.
- Level 1 (intermediate users) heavy explicit use of subagents: Once you discover the prompt pattern of "spawn 5 subagents to analyze your solution, each analyzing a different angle, summarize their findings", it can become addictive. It's not a bad habit per se, but if you're not careful it can drastically overspend your credits.
Level 3 (expert users) extreme multitasking. Just genuinely having 10 worktrees perpetually in parallel and cycling between them in between agent responses. Again, not necessarily bad in itself, but can exponentially conse credits.
What did work pretty well though is MoCa, basically reusing coax/antenna cabling. I get a pretty stable connection roughly equivalent to 2.5G.
I would be happier with my code distributely hosted on every participating node, rather than federating it on my crappy instance.
Also your wallet can be auth + sign so no need for third party auth layers
Less than that actually, considering Rust has its own definition of what "safe" means.
For cables, I think everything converged to cat6a a while ago, which is both reasonably cheap and perfecrly fine for 10G (up to 100m from what I remember)
As for allowing to switch to fiber, that just seems orthogonal again to what these USB NICs are for, not to mention the SFP+ itself is probably more expensive than the NIC shown here...
v4 was built around the idea of multiple free standing networks linked by gateways. v6 was built around the idea of a universal network.
I dont care about what your LAN adress space look like when I'm in my LAN, because we are not in the same v4 network. I am sovereign in my network.
With v6, everyone is effectively in the same network. I have to ask my ISP for a prefix that he will rent me for money even for my LAN. If I want some freedom from said ISP prefix, I am mercifully granted the honor of managing ULA/NAT66 (granted I paid for a fancy router).
Also if I want any kind of privacy, I will have to manage privacy extensions and the great invention of having to use automatically generated, dynamically routed, essentially multiple random IPs per interface. How lucky am I to use such a great new technology.
Seriously v6 was created by nerds in a lab with no practical experience of what people wanted.
v6 adoption is often an all or nothing, because if you run both stacks, you have to ensure they are consistent. While you can reasonably do it on your home LAN, doing it across an entire infrastructure is the worst.
Now you have to make sure all your subnets, routing, VLANs, firewall rules, etc work exactly the same in two protocols that have very little in common.
It is the equivalent of shipping two programs in different languages and maintaining exact feature parity between both at all times.
Neither is IPv6
> To get, basically, the same effect as moving to IPv6
The only thing that IPv6 solves which is of interest to 99.99% of the users is having more adressable space. The rest of IPv6 features are either things that nobody asked for, or things which are genuinely worst compared to IPv4.
I consider the mere fact of enabling IPv6 an unacceptable security risk, as I would now have to make sure my IPv4 and IPv6 firewall stack are perfectly mirroring each other. That would be trivial with IPv4-with-more-bytes, it's a nightmare with IPv6.