HNHacker News
TopNewBestAskShowJobs

nickysielicki

9,596 karma · joined July 28, 2014

email me: hackernews@nslick.com
submissionscomments
nickysielicki··on Nvidia to buy assets from Groq for $20B cash
Groq press release: https://groq.com/newsroom/groq-and-nvidia-enter-non-exclusiv...

> Today, Groq announced that it has entered into a non-exclusive licensing agreement with Nvidia for Groq’s inference technology. The agreement reflects a shared focus on expanding access to high-performance, low cost inference.

> As part of this agreement, Jonathan Ross, Groq’s Founder, Sunny Madra, Groq’s President, and other members of the Groq team will join Nvidia to help advance and scale the licensed technology.

> Groq will continue to operate as an independent company with Simon Edwards stepping into the role of Chief Executive Officer.

> GroqCloud will continue to operate without interruption.

nickysielicki··on When compilers surprise you
The one that always gets me is what Truffle/JRuby was capable of, ten years ago:

https://x.com/chrisgseaton/status/619885182104043520

https://x.com/chrisgseaton/status/619888649866448896

nickysielicki··on When compilers surprise you
Have you considered that maybe Matt isn’t all that surprised by this optimization, but he is excited about how cool it is, and he wants readers of all backgrounds to also be excited about how cool it is, and is just feigning surprise so that he can share a sense of excitement with his audience?

It’s writing for effect.

nickysielicki··on When compilers surprise you
I didn’t write the GP comment. I wouldn’t call this basic and essential, but I would say that compilers have been doing similar loop simplifications for quite some time. I’d expect any mid to senior developer with C/C++ on their resume to at least consider the possibility that the compiler can entirely optimize away a loop.

> In fact, it sounds like something you wouldn't expect them to know.

I’d go a step further, I don’t think anyone, no matter how experienced they are, can confidently claim that optimized assembly will or won’t be produced for a given loop. That’s why the best answer above is, “I dunno”. If performance really matters, you have to investigate and confirm that you’re getting good code. You can have an intuition for what you think might happen, and that’s a useful skill to have on its own, but it’s totally useless if you don’t also know how to confirm your suspicions.

nickysielicki··on When compilers surprise you
Whether they get the question exactly right and can pinpoint the specific compiler passes or algebraic properties responsible for reductions like this is totally irrelevant and not what you’re actually looking for or asking about. It’s a very good jumping point for a conversation about optimization and testing whether they’re the type of developer who has ever looked at the assembly produced in their hotpath or not.

Anyone who dumbly suggests that loops in source code will always result in loops in assembly doesn’t have a clue. Anyone who throws their hands up and says, “I have no idea, but I wonder if there’s some loop invariant or algebraic trick that can be used to optimize this, let’s think about it out loud for a bit” has taken a compiler class and gets full marks. Anyone who says, “I dunno, let’s see what godbolt does and look through the llvm-opt pane” gets an explicit, “hire this one” in the feedback to the hiring manager.

It’s less about what they know and more about if they can find out.

nickysielicki··on macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt
The former will never happen and the latter is a certainty.
nickysielicki··on macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt
This is such a weird project. Like where is this running at scale? Where’s the realistic plan to ever run this at scale? What’s the end goal here?

Don’t get me wrong... It’s super cool, but I fail to understand why money is being spent on this.

nickysielicki··on OpenAI declares 'code red' as Google catches up in AI race
The fundamental problem with bubbles like this, is that you get people like this who are able to take advantage of the The Gell-Mann amnesia effect, except the details that they’re wrong about are so niche that there’s a vanishingly small group of people who are qualified to call them out on it, and there’s simultaneously so much more attention on what they say because investors and speculators are so desperate and anxious for new information.

I followed him on Twitter. He said some very interesting things, I thought. Then he started talking about the niche of ML/AI I work near, and he was completely wrong about it. I became enlightened.

nickysielicki··on CUDA Ontology
If you do this, you forego both backwards and forwards compatibility. You must follow the driver release cadence exactly, and rebuild all of your code for every driver you want to support when a new release happens, or you risk subtle breakage. NVIDIA guarantees nothing in terms of breakage for you.

Probably the worst part of this: for the most part, in practice, it will work just fine. Until it doesn’t. You will have lots of fun debugging subtle bugs in a closed-source black box, which reproduces only against certain driver API header versions, which potentially does not match the version of the actual driver API DSO you’ve dlopened, and which only produces problems when mixed with certain Linux kernel versions.

(I have the exact opposite opinion; people reach too eagerly for the driver API when they don’t need it. Almost everything that can be done with the driver api can be done with the runtime API. If you absolutely must use the driver API, which I doubt, you should at least resolve the function pointers through cudaGetDriverEntrypointByVersion.)

nickysielicki··on Run Nix Based Environments in Kubernetes
What constraints/coordination exists with this, in terms of host driver support? What enforces that Nix does not attempt to use a newer cuda toolkit on a host with an older cuda driver?
nickysielicki··on Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system
> Distributed executor: Inference engines support model parallelism via distributed executors (e.g., Ray [32] and NCCL [9]), whose initialization takes tens of seconds.

I mean, it really shouldn't take tens of seconds for those initialization(s) to occur. There's no good fundamental reason that it should take that long. It's just bloat.

nickysielicki··on Ted Cruz ADS-B Bill Moves to a Vote
The military carve outs for this need to be considered. I realize the desire to prevent accidents like the one that is motivating this, but I live near Fort McCoy and I run an adsb feeder. I hear jets training every day over my head, but they never have their transponders on. This is a good thing as far as I’m concerned, I don’t want to ever be accused of providing our enemies with data on the flight maneuvers they are practicing. I just like seeing the crop dusters do their thing on the map.
nickysielicki··on ADS-B Exposed
I love how you can clearly make out the VFR EAA approach going into Oshkosh from Ripon. It’s only one week out of the year, but there’s so much traffic in that week that it still stands out.
nickysielicki··on QUIC and the end of TCP sockets
> Running in user space offers more flexibility for resource management and experimentation.

I stopped reading here. This isn’t really an essential property of QUIC, there’s a lot of good reasons to eventually try to implement this in the kernel.

https://lwn.net/Articles/1029851/

nickysielicki··on AMD signs AI chip-supply deal with OpenAI, gives it option to take a 10% stake
This isn't stated in any of their press releases.
nickysielicki··on AMD signs AI chip-supply deal with OpenAI, gives it option to take a 10% stake
I wonder what they intend to use for networking, it’s not really clear from this.
nickysielicki··on Docker Was Too Slow, So We Replaced It: Nix in Production [video]
> For a while the only pytorch code that worked on newly released hopper GPUs we had was the Nvidia ngc container, not Pytorch nightly. The upstream ecosystem hadn't caught up yet and Nvidia were adding their special sauce in their image.

I'm sorry to come across as arrogant, but it's really just frustration, because being surrounded by this kind of cargo-culting "special sauce" talk, even from so-called principal engineers, is what drove me to burnout and out of the industry into the northwoods. Furthermore, you're completely wrong. There is no special sauce, you just didn't look at the list of ingredients. There never has been any special sauce.

NVIDIA builds their NGC base containers from open source scripts available on their gitlab instance: https://gitlab.com/nvidia/container-images/cuda

The build scripts for the base container are incredibly straightforward: they add the apt/yum repos and then install packages from that repo.

The pytorch containers are constructed atop these base containers. The specific pytorch commit they use in their NGC pytorch containers are directly linked in their release notes for the container: https://docs.nvidia.com/deeplearning/frameworks/pytorch-rele...

That is:

25.08: https://github.com/pytorch/pytorch/commit/5228986c395dc79f90...

25.06: https://github.com/pytorch/pytorch/commit/5228986c395dc79f90...

25.05: https://github.com/pytorch/pytorch/commit/5228986c395dc79f90...

25.04: https://github.com/pytorch/pytorch/commit/79aa17489c3fc5ed6d...

25.03: https://github.com/pytorch/pytorch/commit/7c8ec84dab7dc10d4e...

25.02: https://github.com/pytorch/pytorch/commit/6c54963f75e9dfdae3...

25.01: https://github.com/pytorch/pytorch/commit/ecf3bae40a6f2f0f3b...

24.12: https://github.com/pytorch/pytorch/commit/df5bbc09d191fff3bd...

24.11: https://github.com/pytorch/pytorch/commit/df5bbc09d191fff3bd...

24.10: https://github.com/pytorch/pytorch/commit/e000cf0ad980e5d140...

24.09: https://github.com/pytorch/pytorch/commit/b465a5843b92f33fe3...

24.08: https://github.com/pytorch/pytorch/commit/872d972e41596a9ac9...

24.07: https://github.com/pytorch/pytorch/commit/3bcc3cddb580bf0f0f...

24.06: https://github.com/pytorch/pytorch/commit/f70bd71a4883c4d624...

24.05: https://github.com/pytorch/pytorch/commit/07cecf4168503a5b3d...

24.04: https://github.com/pytorch/pytorch/commit/6ddf5cf85e3c27c596...

24.03: https://github.com/pytorch/pytorch/commit/40ec155e58ee1a1921...

24.02: https://github.com/pytorch/pytorch/commit/ebedce24ab578036dd...

24.01: https://github.com/pytorch/pytorch/commit/81ea7a489a85d6f6de...

23.12: https://github.com/pytorch/pytorch/commit/81ea7a489a85d6f6de...

23.11: https://github.com/pytorch/pytorch/commit/6a974bec5d779ec10f...

23.10: https://github.com/pytorch/pytorch/commit/32f93b1c689954aa55...

23.09: https://github.com/pytorch/pytorch/commit/32f93b1c689954aa55...

23.08: https://github.com/pytorch/pytorch/commit/29c30b1db8129b5716...

Do I need to keep going? Every single one of these commits is on pytorch/pytorch@main. So when you say:

> For a while the only pytorch code that worked on newly released hopper GPUs we had was the Nvidia ngc container, not Pytorch nightly.

That's provably false. Unless you're suggesting that upstream pytorch continually rebased (eg: force pushed, breaking the worktree of every pytorch developer) atop unmerged code from nvidia, the commit ishes would not match. Meaning all of these commits were merged into pytorch/pytorch@main, and were available in pytorch nightlies, prior to the release of those NGC pytorch containers. No secret sauce, no man behind the curtain, just pure cargo culting and superstition.

nickysielicki··on Docker Was Too Slow, So We Replaced It: Nix in Production [video]
When you’re responsible for supporting people who refuse to receive patches like this one [1], and those same people have the power to page your phone at 11pm on the weekend… you quickly learn how to call a spade, a spade.

[1]: https://patchwork.ozlabs.org/project/ubuntu-kernel/patch/202...

nickysielicki··on Docker Was Too Slow, So We Replaced It: Nix in Production [video]
The modern ML cards are much more raw than people realize. This isn’t a highly mature ecosystem with stable software, there are horrible bugs. It’s gotten better, but there are still big problems, and the biggest problem is that so many people are too stupid to use new releases with the fixes. They stick to the versions they already have because of nothing other than superstition.

Go look at the llama 3 whitepaper and look at how frequently their training jobs died and needed to be restarted. Quoting:

> During a 54-day snapshot period of pre-training, we experienced a total of 466 job interruptions. Of these, 47 were planned interruptions due to automated maintenance operations such as firmware upgrades or operator-initiated operations like configuration or dataset updates. The remaining 419 were unexpected interruptions, which are classified in Table 5. Approximately 78% of the unexpected interruptions are attributed to confirmed hardware issues, such as GPU or host component failures, or suspected hardware-related issues like silent data corruption and unplanned individual host maintenance events.

[edit: to be clear, this is not meant to be a dig on the meta training team. They probably know what they’re doing. Rather, it’s meant to give an idea of how immature the nvidia ecosystem was when they trained llama 3 in early 2024. This is the kind of failure rates you can expect if you opt into using the same outdated software they were forced to use at the time.]

The firmware and driver quality is not what people think it is. There’s also a lot of low-level software like NCCL and the toolkit that exacerbates issues in specific drivers and firmware versions. Grep for “workaround” in the NCCL source code and you’ll see some of them. It’s impossible to validate and test all permutations. It’s also worth mentioning that the drivers interact with a lot of other kernel subsystems. I’d point to HMM, the heterogeneous memory manager, which is hugely important for nvidia-uvm, which was only introduced in v6.1 and sees a lot of activity.

Or go look at the amount of patches being applied to mlx5. Not all of those patches get back ported into stable trees. God help you if your network stack uses an out of tree driver.

nickysielicki··on Docker Was Too Slow, So We Replaced It: Nix in Production [video]
Doesn’t surprise me at all that people who know what they’re doing are building their own images with nix for ML. Tens of millions of dollars have been wasted in the past 2 years by teams who are too stupid to upgrade from buggy software bundled into their “golden” docker container, or too slow to upgrade their drivers/kernels/toolkits. It’s such a shame. It’s not that hard.

Edit: see also the horrors that exist when you mix nvidia software versions: https://developer.nvidia.com/blog/cuda-c-compiler-updates-im...

nickysielicki··on Cloudflare Introduces NET Dollar
Dear whiners: the reason the internet sucks today is because this didn’t already exist. Do you know why Reddit did their horrible redesign and locked down their apps? It wasn’t because you didn’t complain loudly enough, it was because their shareholders were concerned about losing out on profits from data scraping AI companies. Do you know why Twitter can’t be read without logging in? It’s because their shareholders were concerned about losing out on profits from data scraping AI companies. Do you know why you don’t click Quora links? Because they don’t serve you useful results, because they’re concerned about losing profits from data scraping AI companies. Do you see the pattern here?

The open internet died a very long time ago. It’s been dead for years. It’s not coming back unless we figure out a way to make shareholders happy. Paying these companies for the content they host is how that happens.

nickysielicki··on A New Internet Business Model?
> Where did I mention hate?

Substitute whatever adjective you want. You're spreading FUD about the cloudflare boogeyman while ignoring the fact that they have well funded competitors and have no technical mechanism whereby they could lock anyone into their so-called reverse proxy monopoly.

nickysielicki··on A New Internet Business Model?
There's nothing reasonable about your concern. My original reply two levels up explains why: they have competitors and the service they provide is highly fungible.
nickysielicki··on A New Internet Business Model?
So what? I should hate them for that? Cloudflare is really good at what it does. Nobody has to use cloudflare, but people who know what they’re doing choose cloudflare because the service they provide is worth the minuscule price they charge and it solves the massive abuse and performance problems that otherwise plague the internet.

Bing/msn.com failed to displace Google because Google was simply better, not because Google played dirty.

nickysielicki··on A New Internet Business Model?
> Not mentioned: there would be a single gatekeeper for the internet, Cloudflare.

Nothing in their idea challenges the underlying tech behind the internet. Anyone is free to compete in constructing a reverse proxy service with LLM-centric content controls similar to cloudflare, whether that’s AWS WAF or akamai or some new startup.

nickysielicki··on A New Internet Business Model?
Killing the open internet is generally a good thing. Large companies and hostile nation states benefit from the open internet massively, while providing none of it back. The Chinese intranet is not accessible to EU/NA scrapers, but they can read all of our scientific journals. Facebook posts aren’t freely available for you to scrape, but llama is trained on obscure usenet posts and the entire comment history of reddit and hackernews. North Korea has their own linux distribution. Etc.

If the open internet is already dead (and it is already dead), it’s better to accept that reality and silo off the good parts behind paywalls so that people can get paid, rather than to let bad people benefit massively from it while they build their walled gardens. This has been a long time coming.

nickysielicki··on A New Internet Business Model?
Cloudflare has controls already in their dashboard for controlling whether LLMs should be denied responses when querying your site. How they intend to broker payments and selective access isn’t really clear, but you can stop giving your content away for free if you’d like.
nickysielicki··on Systemd can be a cause of restrictions on daemons
Good for you, I'm simply saying that my feelings on the superiority of systemd are informed by having used linux before systemd was around, whereas you insinuated that I had not.

I don't really care to argue about it.

nickysielicki··on Systemd can be a cause of restrictions on daemons
I was in college before Debian adopted systemd, and worked part time as a sysadmin managing hundreds of pre-systemd Debian systems. You’re wrong.
nickysielicki··on Systemd can be a cause of restrictions on daemons
I linked it elsewhere in this thread, but you should really watch this talk, particularly 12:45 through 16:20: https://www.youtube.com/watch?v=o_AIw9bGogo

tl;dr: systemd isn't meant to be an init system, it's meant to manage services, and the alternative world where you don't have a unified system for managing services and events actually sucks.

← PreviousPage 5 of 34Next →