Intel announces Arc B-series "Battlemage" discrete graphics with Linux support
phoronix.com
phoronix.com
At the very least, it's nice to have some decent BUDGET cards now. The ~$200 segment has been totally dead for years. I have a feeling Intel is losing a fair chunk of $ on each card though, just to enter the market.
https://christianjmills.com/posts/intel-pytorch-extension-tu...
https://chsasank.com/intel-arc-gpu-driver-oneapi-installatio...
I quite enjoyed using the CUDA to OneAPI migration tool. Was a bit dodgy at times, but for the most part helped me move a lot of stuff out of the NVIDIA walled garden.
Uh...yes they do? I mean, look at all the VC startups these days like Uber/Doordash/Rivian/WeWork/Carvana/etc. Some of those have been bleeding hundreds of millions per quarter for years and keep getting more pumped in over and over.
But even when it's not for insane VC reasons, companies burn money to get market share all the time. It's a textbook strategy. They just don't do it forever. Eventually, the shareholders can get impatient and decide to bail on the idea.
Assuming they retail at prices Intel are suggesting in the press releases, you maybe here save 40-50 bucks over an ~equivalent NVidia 4060.
I would also argue like others here that with tech like frame gen, DLSS etc, even the cheapest discrete NVidia 40xx parts are arguably 1440p optimized now, it doesn't even need to be said in their marketing materials. Im not as familiar with AMD's range right now, but I suspect virtually every discrete graphics card they sell is "2k optmized" by the standard Intel used here, and also doesn't really warrant explicit mention.
Time marches on, and I become ever more separated from gaming PC enthusiasts.
You say PC gamers at the start of your comment and gaming PC enthusiasts at the end. These groups are not the same and I'd say the latter is largely doing ultrawide, 4k monitor or even 4k TV.
According to steam, 56% are on 1080p, 20% on 1440p and 4% on 2160p.
So gamers as a whole are still settled on 1080p, actually. Not everyone is rich.
Instead, I bought a good 27" 1440p monitor, and you know what? I am not the discerning connoisseur of pixels that I thought I was. Honestly, it's fine.
I will hold out with this setup until I can get a 8k 144hz monitor and a gpu to drive it for a reasonable price. I expect that will take another decade or so.
Most people don't have enough disposable income to make spending that extra amount a reasonable tradeoff (and continuing to spend on upgrades to keep up with their monitor on new games).
In other words, the current tech just isn’t quite there yet, or not cheap enough.
1440p hits a popular balance where it’s more pixels than 1080p but not so absurdly expensive or power hungry.
Eventually 4K might be reasonably affordable, but we’ll settle at 1440p for a while in the meantime like we did at 1080p (which is still plenty popular too).
Tbh now that I think about it I only really need resolution for general usage. For gaming I'm running everything but textures on low with min or max FOV depending on the game so it's not exactly aesthetic anyway. I more so need physical screen size so the heads are physically larger without shoving my face in it and refresh rate.
I run the second monitor off the IGPU so it doesn't even tax the main GPU.
It's doable, the tech is there. But the cost is WAY too high compared to what you get from it in the end.
“2K” is used to denote WQHD often enough, whereas 1080p is usually called that, if not “FHD”.
“2K” being used to denote resolutions lower than WQHD is really only a thing for the 2048 cinema resolutions, not for FHD.
1920x1080 is 1080p.
It doesn't make a whole lot of sense, but that's how it is.
...which will be most likely in better condition than anything that was used for, let's say, gaming...
Untouched video (star wars 8) 4k HDR (60Mb/s) to 1080p at 28fps
No SR-IOV.
No issues with linux. The server did not like the a310, but that is because it is an old dell t430 and it is unsupported hardware. The only thing I had to do was to tweak the fan curve so that it stopped going full tilt.
I guess before you used Nvidia because out of the box support for AMD has existed for ages.
[0] https://store.steampowered.com/hwsurvey/videocard/ - 0.19% share
Video games
These ML AI Macbook people are legit insane.
Desktops and gaming is ugly and complex to them (because lego is hard and macbook look nice unga bunga), yet it is a mass market Intel wants to move in on.
People here complain because Intel is not making a cheap GPU to "make AI" on when that's a market of maybe 1000 people.
This Intel card is perfect for an esports gaming machine running CS2, Valorant, Rocket Leauge and casual or older games like The Sims, GoG games etc. Market of 1 million + right there, CS2 alone is 1mil people playing everyday. Not people grinding leetcode on their macs. Every real developer has a desktop, epyc cpu, giga ram and a nice GPU for downtime and run a real OS like Linux or even Windows (yes majority of devs run Windows)
Or we can keep asking high computers questions about programming.
I agree ML is about to hit (or has likely already hit) some serious constraints compared to breathless predictions of two years ago. I don't think there's anything equivalent to the AI winter on the horizon, though—LLMs even operated by people who have no clue how the underlying mechanism functions are still far more empowered than anything like the primitives of the 80s enabled.
Nvidia had a revenue of $27billion in 2023 - that's about $160 per person per year [0] for every working age person in the USA. And it's predicted to more than double in 2024. If you reduce that to office workers (you know, the people who might actually get some benefit, as no AI is going to milk a cow or serve you starbucks) that's more like $1450/year. Or again more than double that for 2024.
How much value add is the current set of AI products going to give us? It's still mostly promise too.
Sure, like most bubbles there'll probably still be some winners, but there's no way the current market as a whole is sustainable.
The only way the "maximal AI" dream income is actually going to happen is if they functionally replace a significant proportion of the working population completely. And that probably would have large enough impacts to society that things like "Dollars In A Bank" or similar may not be so important.
[0] Using the stat of "169.8 million people worked at some point in 2022" https://www.bls.gov/news.release/pdf/work.pdf
[1] 18.5 million office workers according to https://www.bls.gov/news.release/ocwage.nr0.htm
I'm still not convinced about that. All the """studies""" show 30-60% boost in productivity but clearly this doesn't translate to anything meaningful in real life because no industry laid off 30-60% of their workforce and no industry progressed anywhere close to 30% since chat gpt was released.
It's been released a whole 24 months ago, remember the talks about freeing us from work and curing cancer... Even investments funds which are the biggest suckers for anything profitable are more and more doubtful
I really wasn't interested in computer hardware anymore (they are fast enough!) until I discovered the world of running LLMs and other AI locally. Now I actually care about computer hardware again. It is weird, I wouldn't have even opened this HN thread a year ago.
To have more memory, you have to design a new die with a wider interface. The design+test+masks on leading edge silicon is tens of millions of NRE, and has to be paid well over a year before product launch. No-one is going to do that for a low-priced product with an unknown market.
The savior of home inference is probably going to be AMD's Strix Halo. It's a laptop APU built to be a fairly low end gaming chip, but it has a 256-bit LPDDR5X interface. There are larger LPDDR5X packages available (thanks to the smartphone market), and Strix Halo should be eventually available with 128GB of unified ram, performance probably somewhere around a 4060.
You're right: nobody's doing ML these days. /s
Look at the value of a single website with well over a million users where they publish and run open-weights models on the regular: back in August of 2023, Huggingface's estimated value was $4.5 billion.
B770 was rumoured to match the 16 GB of the A770 (and to be the top end offering for Battlemage) but it is said to not have even been taped out yet with rumour it may end up having been cancelled completely.
I.e. don't hold your breath for anything consumer from Intel this generation better for AI than tha A770 you could have bought 2 years ago. Even if something slightly better is coming at all there is no hint it will be soon.
Hm, i wouldn't consider 200$ low end.
Can you even do ML work with a GPU not compatible with CUDA? (genuine question)
A quick search showed me the equivalence to CUDA in the Intel world is oneAPI, but in practice, are the major Python libraries used for ML compatible with oneAPI? (Was also gonna ask if oneAPI can run inside Docker but apparently it does [1])
Vulkan is especially appealing because you don't need any special GPGPU drivers and it runs on any card which supports Vulkan.
tl;dr GPU's need to transition from being add-in cards to being a sibling motherboard. A sisterboard? Not a daughter board.
It's one of the reasons why ARM Macbooks get great performance/watt, memory being even "closer" than mainboard soldered RAM so getting more of those benefits, though naturally less flexibility.
Intel and AMD internal GPUs can use normal computer RAM. But they are slower for that reason and many others.
This is a graphics card.
-.-
I feel like _anyone_ who can pump out GPU's with 24GB+ of memory that are capable to use for py-stuff would benefit greatly.
Even if it's not as performant as the NVIDIA options - just to be able to get the models to run, at whatever speed.
They would fly off the shelves.
It's going to be a HomePod/AppleTV/Echo/Google Home -style box you set up in a corner and forget about it.
Then your devices in the ecosystem can offload some LLM tasks to that local system for inference, without having to do everything on-device.
Would it though? How many people are running inference at home?
So I would say that Intel's potential market is "everybody who is currently buying nVidia GPUs for compute."
nVidia's stingy consumer RAM choices also seem to be a fairly transparent ploy to create a protective moat around their insanely-high-profit-margin datacenter GPUs. So that just seems like kind of an obvious thing for Intel or AMD to consider tackling.
(Although, it has to be said, a lot of commenters have pointed out that it's not as easy as just slapping more RAM chips onto the GPU boards; you need wider data busses as well etc.)
At some point they should make a stand, that's the whole meta-topic of this thread.
Specifically, waiting ~2 seconds vs ~20 for a code snippet is much more detrimental to my productivity than the time difference would suggest. In ~2 seconds I don't get distracted, in ~20 seconds my mind starts wandering and then I have to spend time refocusing.
Make a GPU that is 50% slower than a 2 generations older mid-range GPU (in tokens/s) but on bigger models and I would gladly shell out 1000+$.
So much so that I am considering getting a 5090 if nVdia actually fixes the connector mess they made with 4090s or even a used v100.
Running a specialist model makes more sense on small devices.
Well informed gamers know Intel's discrete GPU is hanging by a thread, so they're not hoping on that bandwagon.
Too small for ML.
The only people really happy seem to be the ones buying it for transcoding and I can't imagine there is a huge market of people going "I need to go buy a card for AV1 encoding".
Nvidia is trash tier in terms of support and only recently making serious steps to actually support the platform.
AMD went all in nearly a decade ago and it's working pretty well for them. They are mostly caught up to being Intel grade support in the kernel.
Meanwhile, Intel has been doing this since I was in college. I was running the i915 driver in Ubuntu 20 years ago. Sure their chips are super low power stuff, but what you can do with them and the level of software support you get is unmatched. Years before these other vendors were taking the platform seriously Intel was supporting and funding Mesa development.
They have been improving because getting CUDA running on the platform in datacenters required some level of cooperation. They eventually realized this had to happen regardless for AI in the datacenter and so eventually they got a kernel module starting to be added. It's not really there yet and it doesn't run the actual driver, but it gets a pathway to booting Nvidia with signed drivers among other things.
The end result is our support forums are still full of Nvidia users. Go to any support forum or Reddit related to Linux and view the comments and press CTRL+F for Nvidia. It's so common we have to just tag/bin these posts and move on because there are literally man years of time being wasted on this.
> AMD on the other hand gives lousy support.
I can open a bug right now on both GitHub and the LKMS directly with AMD engineers. When I have a question about ROCm I can ask them and get a quick response. These guys are in Reddit threads talking to ROCm users and giving them guidance on how to modify their consumer grade Radeon cards to use the datacenter grade drivers that aren't adapted to consumer facing cards yet.
The one thing we for sure agree on is that Intel doesn't have any of this BS. Their stuff just works and their pipeline for getting anything fixed is also well optimized.
Had this been available a few weeks ago I would have gone through the pain of early adoption. Sadly it wasn't just an upgrade build for me, so I didn't have the luxury of waiting.
What do you mean by this - I assume you mean too small for SoTA LLMs? There are many ML applications where 12GB is more than enough.
Even w.r.t. LLMs, not everyone requires the latest & biggest LLM models. Some "small", distilled and/or quantized LLMs are perfectly usable with <24GB
Still...tangibly cheaper than even a 2nd hand 3090 so there is perhaps a market for it
If Intel's stats are anything to go by, League runs way better than it did on the last generation and it's the only game that has had issues on the last-gen that's still left running on DX9, CS:GO was another notable one but CS2 has launched since and the game has moved to DX12/VK. This was, literally, the biggest issue they had - drivers were also wonky but they seem to have ironed that out as well.
Ubuntu 24.04 couldn't even boot to a tty with the Nvidia Quadro thing that came with this major-brand PC workstation, still under warranty.
Why would that matter? You buy one GPU, in a few years you buy another GPU. It's not a life decision.
The game devs are going to spend all their time & effort targetting amd/nvidia. Custom code paths etc.
It's not a one size fits all world. OpenCL etc abstraction are good at covering up differences, but not that good. So if you're the player with <10% market share you're going to have an uphill battle to just be on par.
From my experience they target NVidia and Consoles. AMD might get a look at the code just before release if they notice any big problems.
I'd be surprised if many Gamedevs even pick up the phone for an Intel GPU developer.
They do well compared to AMD/Nvidia at that price point.
Is it a market worth chasing at all?
Doubt.
Intel's customers are 3rd party Cpu assemblers like Dell & HP. Many corporate bulk buyers only care if 1-2 of the apps they use are supported. The lack of wider support isn't a concern.
In particular, intel just needs to support vfio and it’ll be huge for homelabs.
there is a niche for this, though it remains to be seen if it'll be profitable enough for a large org like Intel
I am hoping these are open in such a manner that they can be used in OpenBSD. Right now I avoid all hardware with a Nvidia GPU. That makes for somewhat slim pickings.
If the firmware is acceptable to the OpenBSD folks, then I will happly use these.
I don't think that's a super fair shake? Intel iGPUs have been around for a while if you had a laptop chip or iGPU-enabled desktop chip. They've supported Linux just fine for ages, and will fill any non-3D application you might have.
And Nvidia chips are quite good on Linux nowadays - Wayland has been very usable since the 535-series drivers and nearly flawless since 550. You're right to be apprehensive about proprietary GPU hardware but I think there are plenty of options on the table right now.
For power, it's 190W compared to 4060's 115 W.
EDIT: from [1]: B580 has 21.7 billion transistors at 406 mm² die area, compared to 4060's 18.9 billion and 146 mm². That's a big die.
If we use the numbers from the preview:
| |Arc A770|Arc B580|RTX 4060|
|--------|--------|--------|--------|
|Process |N6 |N5 |N5 |
|Die Size|406mm^2 |272mm^2 |159mm^2 |
|Trans. |21.7B |19.6B |18.9B |
|Mem Bus |256 bit |192 bit |128 bit |
|TDP |225W |190W |115W |
|~Perf |90% |110% |100% |
- Nvidia has better layout. - Intel is using higher performance (larger) transistor libraries or layout in order to hit the higher boost frequencies (2800 vs 2460). - Wider bus interface takes up more space. - The B580 may have 1 render slice and 64-bits of memory bus disabled, and they're not including those transistors in the count, but they still take up area.
[0]: https://www.techpowerup.com/review/intel-arc-b580-battlemage...
1. There are dummy transistors included in the design for various manufacturing reasons, and there's no standard as to whether those are included in the numbers.
2. SRAM cells in particular are highly optimized and the amount of SRAM (e.g. cache memory) on the chip versus logic will have a big influence on the transistor density.
He said he's still asking around the engineering team for a more complete explanation but that's the supposition.
https://www.techpowerup.com/review/intel-arc-b580-battlemage...
These numbers seem bit more believable
It's a real shame, the single slot a380 is a great performance for price light gaming and general use card for small machines.
Also, I don't think you'll find many mainboards from 2006 supporting it. It may have been standardized in 2006, but a quick online search leads me to think that even on x86 mainboards it didn't become commonly available until at least 2020.
Coffee Lake also didn't really support ReBAR either, also 2018.
Another issue is that not every GPU actually supports ReBAR, I'm reasonably certain the Nvidia drivers turn it off for some titles, and pretty much the only vendor that reliably wants ReBAR on at all times is Intel Arc.
I also personally wouldn't say that Sandy Bridge is very usable with a modern GPU without also specifying what kind of CPU or GPU. Or context in how it's being used.
I cling onto my old hardware to limit ewaste where I can. I still gave up on my old sandybridge machine once it hit about a decade old. Not only would the CPU have trouble keeping up, its mostly only PCIe 2.0. A few had 3.0. You wouldn't get the full potential even out of the cheapest one of these intel cards. If you are putting a GPU in a system like that I can't imagine even buying new. Just get something used off ebay.
What I'm also interested in is whether they'll require a PCIE 4.0 motherboard or if a 3.0 x16 slot is fine.
Also whether their software properly supports VR now...
I haven't regretted the purchase at all.
IMO you want those frames getting rendered as close to the monitor as possible, and you'd probably have a better time with lower fidelity graphics rendered locally. You'd also get to keep gaming during a network outage.
I've tried game streaming under the best possible conditions (<1ms network latency) and it still feels a little off. Especially shooters and 2D platformers.
The only concern is how well the new Intel drivers work (full support for DX12) with older titles which are continuously being improved (for DX11, 10, and some for 9 others via emulation).
There's likely some deep discounting of Intel cards because of how bad the drivers were at launch and the prices may not stay so low once things are working much better.
Fortunately, having their Linux drivers be (mostly?) open source makes a purchase seem less risky.
It is in the (minor) sense that I'd rely on Intel for warranty support, driver updates (if closed source), and firmware fixes.
But I agree with your main point that the worst-case downside isn't that big of a deal.
These are obviously Windows-specific issues that don't come up at all in Linux, where all that Direct3D headache is taken care of by DXVK. Amusingly a big part of Intel's efforts to improve D3D performance on Windows has been to use DXVK for many titles.
Battlemage is supposed to fix all these architectural issues. EU in Xe2 is now SIMD16 (which is why the number of EUs per Xe2 core is halved from that of Xe1), and they've added all the previously software-emulated instructions, including Execute Indirect, so in theory Battlemage should be in a much better position in game compatibility side of things.
On Linux side of things, lacking sparse residency support in i915 also contributes to game compatibility[1] (though this is now available under Mesa 24). This is something the new xe driver is supposed to fix, but it's still a long way to go until it's actually usable.
I've had a small sound latency issue forever; most visible with YouTube videos, the first half-second of every video is silent.
I picked this card up for about $120 less than the GTX 4060. Wasn't a terrible decision.
Presumably graphics cards optimised for hairdressers and telephone sanitisers?
Other exciting tests will include things like fan control, since that’s still an issue with Arc GPUs.
Should make for a fun blog post.
Very happy with my A770. Godsend for people like me who want plenty VRAM to play with neural nets, but don't have the money for workstation GPUs or massively overpriced Nvidia flagships. Works painlessly with linux, gaming performance is fine, price was the first time I haven't felt fleeced buying a GPU in many years. Not having CUDA does lead to some friction, but I think nVidia's CUDA moat is a temporary situation.
Prolly sit this one out unless they release another SKU with 16G or more ram. But if Intel survives long enough to release Celestial, I'll happily buy one.
My experience on WH40K DT has taught me that upscaling is absolutely vital for a reasonable experience on some games.
This strikes me as a bit of a sad state of affairs. We've moved beyond a Parkinson's law of computational resources –usage by games expands to fill the available resources– to resource usage expanding to fill the available resources on the highest end machines unavailable for less than a few thousand dollars... and then using that to train a model to simulate by upscaling higher quality or performance on lower end machines.
A counterargument would be that this makes high-end experiences available to more people, and while in the individual case, I don't buy that that's where the incentives it creates are driving the entire industry.
To put a finer point on it: at what percentage of budget is too much money being spent on producing assets?
What a time to be alive. Our most advanced technology is used to cheat on homework and play video games.
But surely it's easy enough to compete on video ram - why not load their GPUs to the max with video ram?
And also video encoder cores - Intel has a great video encoder core and these vary little across high end to low end GPUs - so they could make it a standout feature to have, for example, 8 video encoder cores instead of 2.
It's no wonder Nvidia is the king because AMD and Intel just don't seem willing to fight.
I think it's the right call since there isn't much competition in GPU industry anyway. Sure, Intel is far behind. But they need to start somewhere in order to break ground.
Strictly speaking strategically, my intuition is that they will learn from this, course correct and then would start making progress.
> "They need to start somewhere in order to break ground"
Intel has big problems and it's not clear they should occupy themselves with this. They should stabilize, and the most plausible way to do that is to cut the weak parts, and get back to what they were good at - performant secure x86_64 CPUs, maybe some new innovative CPUs with low consumption, maybe memory/solid state drives.
"there is not much money in it"?
WTF?
Intel tried to get into GPU-like products 14 years ago. They promote their consumer Arc GPUs since 2022, and still almost nobody wants them. Is there a datacenter Intel GPU that some business wants?
The big money is flowing NVIDIA's way, and even if Intel can make a GPU, it won't be able to divert a big part of the flow, similarly to AMD.
However, this is going to go on clearance within 6 months. Good for consumers, bad for Intel.
Also keep in mind for any ML task Nvidia has the best ecosystem around. AMD and Intel are both like 5 years behind to be charitable...
How was driver support for their A-series?
They likely won't need to do the same discovery and fixing for B-series as they've already dealt with it.
Not a huge fan of the numbering system they've used. B > A doesn't parse as easily as 5xxx > 4xxx to me.
IIRC that was one of the original goals of geohot's tinybox project, though I'm not sure exactly where that evolved
Their dedication to Linux Support, combined with their good pricing makes this a potential buy for me in future versions. To be frank, I won't be replacing my 7900 XTX with this. Intel needs to provide more raw power in their cards and third parties need to improve their software support before this captures my business.
ah well. pretty sure it'll do for my needs.
Based on scaling by XMX/engine clock napkin math, the B580 should have 230 FP16 TFLOPS and 456 GB/s MBW theoretical. At similar efficiency to LNL Xe2, that should be about pp512 ~4700 t/s and tg128 ~77 t/s for a 7B class model. This would be about 75% of a 3090 for pp and 50% for tg (and of course, 50% of memory). For $250, that's not too bad.
I do want to note a couple things from my poking around. The IPEX-LLM [1] was very responsive, and was able to address an issue I had w/ llama.cpp within days. They are doing weekly update releases, so that's great. The IPEX stands for Intel Extension for PyTorch [2] and it is a mostly drop-in for PyTorch: "Intel® Extension for PyTorch* extends PyTorch* with up-to-date features optimizations for an extra performance boost on Intel hardware. Optimizations take advantage of Intel® Advanced Vector Extensions 512 (Intel® AVX-512) Vector Neural Network Instructions (VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel Xe Matrix Extensions (XMX) AI engines on Intel discrete GPUs. Moreover, Intel® Extension for PyTorch* provides easy GPU acceleration for Intel discrete GPUs through the PyTorch* xpu device."
All of this depends on Intel oneAPI Base Kit [3] which has easy Linux (and presumably Windows) support. I am normally an AUR guy on my Arch Linux workstation, but those are basically broken and I had much more success installing oneAPI Base Kit (w/o issues) directly in Arch Linux. Sadly, this is also where there are issues some of the code is either dependent on older versions of oneAPI Base Kit that are no longer available (vLLM requires oneAPI Base Toolkit 2024.1 - this is not available for download from the Intel site anymore) or in dependency hell (GPU whisper simply will not work, ipex-llm[xpu] has internal conflicts from the get go), so it's not all sunshine. On average, ROCm w/ RNDA3 is much more mature (while not always the fastest, most basic things do just work now).
[1] https://github.com/intel-analytics/ipex-llm
[2] https://github.com/intel/intel-extension-for-pytorch
[3] https://www.intel.com/content/www/us/en/developer/tools/onea...
I'm guessing their marketing department isn't known as the "A-team".
Because, no. It's not as simple as that.
NVIDIA has a complete ecosystem now. They have cards. They have cards of cards (platforms), which they produce, validate and sell. They have NVLink crossbars and switches which connects these cards on their card of cards with very high speeds and low latency.
For inter-server communication they have libraries which coordinate cards, workloads and computations.
They bought Mellanox, but that can be used by anyone, so there's no lock-in for now.
As a tangent, NVIDIA has a whole set of standards for pumping tremendous amount of data in and out of these mesh of cards. Let it be GPU-Direct storage or specialized daemons which handle data transfers on and off cards.
If you think that you can connect n cards on PCIe bus and just send workloads to them and solve problems magically, you'll hurt yourself a lot, both performance and psychology wise.
You have to build a stack which can perform these things with maximum possible performance to be able to compute with NVIDIA. It's not just emulating CUDA, now. Esp., on the high end of the AI spectrum (GenAI, MultiCard, MultiSystem, etc.).
For other lower end, multi-tenant scenarios, they have card virtualization, MIG, etc. for card sharing. You have to complete on that, too, for cloud and smaller applications.
https://github.com/ryao/llama3.c
Inference workloads are easy to parallelize to N cards with minimal connectivity between them. The Nvlink crossbars and switches just are not needed.
In particular, inference can be divided into two distinct phases, which are input processing (prompt processing) and output generation (token generation). They are remarkably different in their requirements. Input processing is compute bound via GEMM operations while output generation is memory bandwidth bound via GEMV operations. Technically, you can do the input processing via GEMV too by processing 1 token at a time, but that is slow, so you do not want to do that. Anyway, these phases can be further subdivided into the model’s layers. You can have 1 GPU per layer with the logits passing from GPU to GPU in a pipeline. The GPUs just need the layer’s weights and the key-value cache for all of the tokens in that layer in memory to be able to work effectively. For llama 3.1 405B, there are 126 layers, so that is up to 126 GPUs.
That is of course slightly slower than if you just had 1 GPU with an incredible amount of VRAM, but you can always have more than one query in flight to get better than 1 GPU’s worth of performance from this pipeline approach. There are other ways of doing parallelization too, such as having output processing use GEMM to do multiple queries in parallel. This would be what others call batching, although I am only interested in doing 1 query at a time right now, so I have not touched it.
In essence, you can connect n cards on PCIe and have them solve inferencing problems magically, with the right software. Training is a different matter and I cannot comment on it as I have not studied it yet.
The training can come after, with some inference and runtime optimizations on the software stack.
I think the high memory local inference stuff is going to come from "AI enabled" cpus that share the memory in your computer. Apple is doing this now, but cheaper options are on the way. As a shape its just suboptimal for graphics, so it doesn't make sense for any of the gpu vendors to do it.
In this case, the die I/O limits precludes more than a reasonable number of DDR channels.
High speed, low latency server grade DDR5 is around $800-$1600 for 128GB. Triple that for $2400 - $4800 just for the memory. Still need the GPUs/APUs, card, VRMs, etc.
Even the nVidia H100 with "only" 94GB starts at $30k...
It is why you can only see GPUs with 24GB of memory at the moment.
HBM2 can handle 64GB ( 4 x 8GB Stack ) ( Total capacity 128GB )
HBM3 can handle 192GB ( 4 x 24GB Stack ) ( Total capacity 384GB )
You can not do this with GDDR6.
https://www.qualcomm.com/news/onq/2023/11/introducing-qualco...
I have no idea how much it costs. They do not sell it via PC parts channels.
The deeper problem is that the market for this is probably incredibly niche.
Beyond that, you'd have to move to GDDR7 (which has 24Gb/3GB chips incoming) or to HBM stacks, but at that point you're well beyond a "basic GPU". I think the only way you could get to 128GB would be either using regular (LP)DDR or HBM.
Note, Apple M chips have weak GPUs with decent MBW and large memory capacities (up to 192GB @ 800GB/s for an M2 Ultra, launched mid 2023) and have not been a major CUDA threat so I don't think your hypothesis actually stands up.
besides the clear limitation of the memory technology they are using compared to the nvidia's enterprise solution, for such large GPU chips that could really make use of such memory, they need to make binning possible by selling cut-down versions of them as well.
nvidia can pull this off because they can sell lower-end chips at the same time. intel is barely making a dent in sales, and making a high-end chip will only be very risky, at the cost of potentially benefiting a niche crowd.
> put them as a major CUDA threat. that is a software/ecosystem problem, which hardware alone cannot solve. for all the devs that use Macs, even in AI it is only about inference at the moment. nobody is coming at CUDA for training for the near future. amd tried and failed plenty already.
Rather than go all in with 128GB, they could test the waters easily with a cheap 32GB offering and take ot from there.
I do think one challenge is, AFAIK with most GDDR5/6 there's a density issue that requires either larger memory bus paths or other additional complexity to support large sizes.
That said, the lack of even a 16GB variant is sus.
I'll take some copium in that maybe they're trying to solve the 'size' issue somehow and are just making sure whatever system they use isn't gonna be an i820 MTH debacle before they pull the trigger on announcing it.
But obvoiusly they don't.
And for reasons: NVidia has worked on CUDA for ages, do you believe they just replace this whole thing in no time?
The lack of quantified stats on the marketing pages tells me Intel is way behind.