HNHacker News
TopNewBestAskShowJobs

wtallis

22,096 karma · joined September 20, 2009

submissionscomments
wtallis··on Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
Sure, if you want to make a comparison where the price tags aren't the same order of magnitude, then a recent server is obviously going to be powerful. But since the baseline of this comparison is a laptop and several Thunderbolt SSDs, the kind of servers or workstations with 1.5–2TB of RAM that you can reasonably compare against would have to be the really old ones, barely new enough to support that much total RAM.

And despite the theoretically high memory bandwidth of recent EPYC CPUs, approximately nobody who can afford one is doing LLM inference on them.

wtallis··on Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
ASIC vs GPU doesn't make a ton of difference when both are relying on commodity DRAM; in that sense, LLMs are more like the anti-ASIC cryptocurrencies. But the actually interesting ASICs are the ones that ditch the commodity discrete DRAM chips. They lose out on the memory density and thus struggle to scale up to the largest models, but for what does fit onto a Cerebras wafer or a Taalas chip, the speed is phenomenal. They have a real shot at securing the "smart enough, and really fast" segment of the market.

And it seems more plausible to me that an ASIC architecture rather than GPUs would be able to best make use of something like wafer-bonded custom memory to approach the density of discrete DRAM while retaining the extremely high bandwidth that comes with arbitrarily wide interfaces and minimal PHYs.

wtallis··on Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
To be fair, nobody has upgradable memory in any system that has enough memory bandwidth and compute power to run LLMs with decent performance. It might be interesting to compare against some decade-old x86 server or workstation stuffed full of LRDIMMs to reach 1.5–2TB of RAM, but the bandwidth would be only slightly faster than a desktop today with high-end DDR5: nowhere close to GPU bandwidth. So performance would still suck.

Designing for extreme expandability comes with pretty steep tradeoffs.

wtallis··on Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
To the extent that cryptocurrency moved off ASICs, it was because of interest shifting to different cryptocurrencies that were specifically designed to be harder to mine on an ASIC than Bitcoin's compute-heavy, memory-light hashing.

I'm not sure there's any reason to expect a similar shift from LLMs. The hardware used for training doesn't dictate what hardware needs to be used for inference, and nobody's going to design an LLM architecture with an overt intention to make it better suited to GPUs and hard to target with ASICs.

wtallis··on LG TVs caught spying even when offline or on standby
Isn't messing with TTLs a pretty obvious thing to do when communicating covertly? And wouldn't that kind of packet inspection only catch cases of TVs literally acting as packet-level routers, but not catch application-layer shenanigans like aggregating statistics in the exit node so that all the data going over your own network appears to be originating from the same device?

It's still implausible, but thoroughly verifying that your TV isn't spying on you isn't trivial.

wtallis··on Cheap Desktop 400GbE Switch MikroTik CRS804-4DDQ-HRM Review
I think this is the distinction between pipeline parallelism (first N layers on machine 1, next N layers on machine 2, etc.) vs tensor parallelism where all machines contribute to the same matrix multiplication operation within the same layer. Pipeline parallelism doesn't need exotic high speed networking, but also doesn't deliver the kind of speed increase that tensor parallelism provides.
wtallis··on Cheap Desktop 400GbE Switch MikroTik CRS804-4DDQ-HRM Review
Please be careful with units. There are high-end consumer SSDs that will deliver data at 14+GB/s, which is much higher than 25Gb/s.
wtallis··on Can I opt out of my input or output data being used for training?
It's also simply bad writing regardless of what meaning the reader takes from it. "Enabled by default" or "Allowed by default" are much clearer and more natural phrasings without any connotations of the user having taken an action to express a choice.
wtallis··on Firefox's AI Switch Is Off. Telemetry Isn't
At some point, the pervasive unethical behavior of having telemetry on by default instead of opt-in, not just in proprietary software but also in open-source software and even from Mozilla itself becomes overwhelming enough that "telemetry=bad" becomes a reasonable summary of the situation. We need a new word specifically for ethical opt-in data collection.
wtallis··on Hang on to Your Firefox
Sync and container tabs aren't on the list of features I recall seeing people complain about.
wtallis··on Google Has Removed MV2 Extensions from the Chrome Web Store, Including UBO
Having what you consider to be a good excuse doesn't mean it didn't happen. Multiple times.
wtallis··on Google Has Removed MV2 Extensions from the Chrome Web Store, Including UBO
Mozilla has had their share of mass extension deprecation and breakage.
wtallis··on Google Has Removed MV2 Extensions from the Chrome Web Store, Including UBO
Ad blocking has always been a safety issue.
wtallis··on Google Has Removed MV2 Extensions from the Chrome Web Store, Including UBO
I agree that on a large scale, Chrome made browsing safer for the general public at the time. But for me it made things slower, use more RAM, waste more bandwidth on ads, didn't provide meaningful security improvements, and was a big downgrade in privacy. The only time I've been successfully attacked through a browser was when a Chrome extension sold out and became adware. And Google is now deciding for a frighteningly large part of the public exactly how much security and privacy they should get.
wtallis··on Google Has Removed MV2 Extensions from the Chrome Web Store, Including UBO
The "Just buy a new computer" excuse doesn't actually make the tradeoff no longer a tradeoff. The cost is still real, you're just putting a price tag on it (then ignoring the price, along with any alternative tradeoffs).
wtallis··on Google Has Removed MV2 Extensions from the Chrome Web Store, Including UBO
Chrome's multiprocess sandboxing wasn't unambiguously good. It carried a steep cost in overhead.

Back when Chrome was new, you could run Firefox with a decent selection of add-ons to get security benefits of similar magnitude through different mechanisms. The NoScript extension was at the forefront, providing a lot of functionality way beyond just blocking JavaScript per domain. And Firefox with extensions always had better privacy controls.

wtallis··on Why open source rocks – a new SM750 (Silicon Motion GPU) HDMI Driver
It doesn't really do much in the way of processing (https://www.siliconmotion.com/download/3PS/a/SM750_PB_EN_201...); it's pretty much just a dumb framebuffer. You can wire it up to output over an HDMI connector, but with the features it supports you're really just getting a single-link DVI signal.
wtallis··on Show HN: I missed the moving blocks, so I built a real Linux disk defragmenter
You cannot directly inspect the degree of fragmentation, because it has less to do with being contiguous in the Logical Block Address (LBA) space and more to do with having been written at the same time. To properly defragment a file on a SSD, you pretty much need to sequentially re-write the whole file in one go, to a newly-allocated part of the drive's LBA space.
wtallis··on Processing in Memory: DRAM Is About to Do Math
In case you haven't been paying attention, DRAM and NAND are also pretty expensive these days, relative to where they were a year ago.

The drastically different scaling trends for logic and SRAM on the same die are important, but I'm not sure how relevant it is to LLM inference. Cerebras has already shown that the amount of SRAM you can get out of a wafer is an absolute pittance compared to what frontier LLMs require; a multi-million dollar machine with ~40GB of SRAM is not a good solution for big LLMs no matter how fast it may be for small models. Denser forms of memory are almost mandatory for doing LLM inference on the largest models, and it seems pretty clear that the most cost-effective architectures we can build today for LLM inference would not be dominated by SRAM. You'll still have SRAM caches, but it doesn't make sense as the primary storage for the entire model's weights.

wtallis··on Processing in Memory: DRAM Is About to Do Math
I'm going to ignore the first half of your comment, because it really only serves to disclose the limits of your knowledge.

The problem with your rambling and speculation is that you're not making any attempt to think about what conditions would be necessary for a technology to succeed in the market and gain widespread adoption across many market segments. I stand by my assertion that it is impossible for compute-in-memory chips to entirely replace commodity DRAM, especially in any market segment where the compute portion of those memory chips is not critically valuable. Any remotely plausible, useful compute-in-memory design will have significantly lower capacity and higher cost than pure commodity DRAM; this is not a low-overhead added feature. Any market segment where the compute capabilities of compute-in-memory were so marginally valuable that a non-trivial number of the customers wanted to disable the compute is a market segment where everybody would still be buying mostly ordinary DRAM.

wtallis··on Processing in Memory: DRAM Is About to Do Math
I wonder if we might eventually see stacking of NAND, logic, and DRAM dies together so each can still be fabbed on their specialized process. Getting all three on one wafer is extremely impractical, but having them all connected with TSVs or other bonding might be feasible, if the density of the respective dies works out right to give you the right ratio of storage for weights, working memory, and compute.
wtallis··on Processing in Memory: DRAM Is About to Do Math
You seem to think that "compute in memory" would mean the disappearance of commodity DRAM parts and their replacement with parts that had compute capabilities. This is impossible. Adding compute to memory will make it more expensive, so it will get zero adoption outside of the niches where it actually proves to be useful and valuable. Nobody is going to be putting compute in memory chips onto standard consumer DIMMs.
wtallis··on GUIs should be fully keyboard-driven
It sounds like your complaint might be that it works poorly if you half-ass the localization process and assume that literal translation of strings is most or all of what localization means.
wtallis··on Humanity has the debate about AI consciousness backwards
I don't think that argument holds water, because the scope of what can be reasonably approximated by matrix multiplication is extremely broad (especially when the matrix multiplication is also paired with a source of randomness).
wtallis··on Compromising Signal's Contact Discovery Enclave (SGX)
Is this or any other SGX exploit a consequence of Intel processors not being open source?
wtallis··on Apple introduces M6 and M5 Ultra
I wonder if the dislike comes from using third-party keyboards with excessively wide spacebars that make the modifier keys genuinely hard to reach with a thumb.
wtallis··on Xiaomi: New CPU matches Apple cores single threaded, much faster multithreaded
It's been a long time since anyone tried to actually ship a chip with different instruction set support enabled across its cores, but Intel has spent the past several years shipping consumer parts where they had to disable some features (AVX-512) on their P-cores to bring them down to the level of what their E-cores support, because they didn't plan ahead for doing a heterogeneous chip with those two microarchitectures. Toggling off some instructions in microcode for some cores is quite different from having two microarchitectures that are each designed to implement the same instruction set but with different performance and power characteristics.
wtallis··on Xiaomi: New CPU matches Apple cores single threaded, much faster multithreaded
Did Intel ever ship any heterogeneous server parts? I'm not aware of any, and it looks like even the Raptor Lake desktop silicon sold under the Xeon brand only had P-cores enabled.

The worst of Intel's processor core asymmetry was with Meteor Lake and Arrow Lake, where the two LP-E cores on the IO die had no L3 cache and really bad DRAM latency despite being directly adjacent to the DRAM controller, resulting in those two cores combined offering about as much performance as one of the E-cores on the CPU chiplet. And they used the same IO die across both generations, while updating the microarchitecture of the CPU chiplet, so things diverged even more. And because the performance of the LP-E cores was so bad, Windows wouldn't schedule any threads on them unless it was trying to operate exclusively on those LP-E cores with the CPU chiplet powered down entirely—but those two LP-E cores were still reported to software in the total thread count, so heavy multithreaded applications would launch two more threads than could actually be scheduled at once.

With Panther Lake, Intel still has three tiers of CPU cores, but the performance gap between E cores and LP-E cores is down to only about 10%.

wtallis··on Xiaomi: New CPU matches Apple cores single threaded, much faster multithreaded
What degree of OS-indepedence do you think is reasonable to expect from a consumer-facing benchmark suite like Geekbench?
wtallis··on Civic Hygiene – avoid building technologies that could be used by a police state (2013)
Generally, yes. Most telemetry systems in software are not producing enough actionable insight to justify the privacy invasion. Spying on your customers on an opt-out basis just in case you one day come up with a use for that data is a widespread practice but clearly unethical, and leaves you (or whoever acquires your employer) in the position of being tempted to use the data for worse things than just deciding what bugs and features to prioritize.
← PreviousPage 3 of 34Next →