Apple Is Developing AI Chips for Data Centers, Seeking Edge in Arms Race | Hacker News Reader

Apple Is Developing AI Chips for Data Centers, Seeking Edge in Arms Race

wsj.com

44 pointsbreck53 comments

Obviously nobody is competing with NVIDIA right now given their strong CUDA+Mellanox advantage. But that second place is a very important spot.

I think if Apple sold to businesses, it would be really bad news for AMD.

For comparison, take an MI300A which is AMDs H100 competitor. It’s under half the price ($20k) of an H100 (~$30-50k) with more HB3 memory. It maxes out at something like 128GB of memory. I can’t find the price of a GH100 in todays market.

Then there’s Apple. An M2 Ultra Mac Studio is <$7k for 192GB of memory. Granted it’s much slower memory, but that’s not a huge factor for many workloads.

So this M2 Ultra machine is 14% of the H100 and 35% of the MI300A. It has more memory and a faster per-core processor that consumes 40% of the power of the MI300A.

Now of course there aren’t good benchmarks to compare the exact loadouts and configurations here, and like I mentioned the memory is slower LPDDR5 instead of HBM3.

but even then, you can get multiple Mac Studios for the closest competitor. It’ll cost you significantly less to run. The extreme memory bandwidth of HBM3 is not unique to anyone and the price difference would likely more than cover it.

So then there’s the software moat. NVIDIA has CUDA and then after them, Apple seemingly has the widest support of ML libraries. AMD has improved a lot with Rocm but they’re still lagging behind Apple and way behind NVIDIA.

So anyway, if Apple decides to bring this to market, I would be very worried for anyone vying for second place. The biggest issues will be whether macOS is a turnoff (doubt it) and whether Apple can do enterprise (remains to be seen).

Tensorflow seems to run fine on Google TPUs.

Yes but Google don’t sell their hardware to others, so there’s no pricing information to compare it with.

The mi300x is AMDs h100/200 competitor.

I know. But the M2 Ultra is the MI300A’s competitor. I couldn’t find the pricing for the GH100, but used the H100 as a standin since it’s already so much more expensive to not really affect the point I was making.

Does Apple ever compete on price?

Yes but only if you shop to match the same criteria.

E.g a MacBook Pro may seem overpriced if you compare to average laptop costs but if you shop within laptops with the same display, battery life and performance unplugged, it is very fairly priced.

Similarly, Vision Pro seems expensive compared to average consumer headsets, but if you match its capabilities (eye tracking, high resolution display, etc) the next competitor are multiples of its price.

> The biggest issues will be whether macOS is a turnoff

Could Asahi Linux become relevant here?

I would think not. The GPU is a bit part of the equation for compute , and there’s very little in the way of ML libraries targeting the standard graphics APIs available on Linux.

Is that true? Linux is the number one choice for training and inferencing deployment, somebody must be doing something right. It's not like OpenAI was training their LLMs using CoreML, Apple isn't even a contender in the game. They were late to supporting basic industry-standard libraries like Tensorflow and Pytorch, and they still don't commit to an Open, cross-platform alternative.

Apple is the #1 loser from the lack of standardization here, and you're trying to make it out as a Linux problem. Imagine if someone from the industry could hear you talking right now.

iOS developer here, CoreML is used for edge deployments in the ML space. Just not training. Obviously linux is king but that's for server side my dude.

"edge deployment" by which you mean client-side application-specific inference. That's not bad, but it's also table-stakes when ONNX/Tensorflow exists and obviously isn't analogous to the server products Nvidia is shipping today. You can "deploy" CoreML the same way Android developers deployed Tensorflow for edge compute 5 years ago, but that wasn't even interesting when people demoed it on Raspberry Pi. Apple is behind.

> Obviously linux is king but that's for server side my dude.

But isn't that what we're talking about? The submission title is "Apple is developing AI chips for data centers".

Right, "developing"

It's a tiny bit ignorant to even compare consumer memory to HBM. Anyone can get 1TB for their server for less than Ultra chips. The reason you go HBM is precisely because of bandwidth needs.

I specifically call out the memory difference though precisely because I know they’re different.

But I do not believe the cost for 192GB of HBM3 + any related changes to the memory system on the chips would be $13k, which is the difference between the Mac Studio and the MI300A.

It really feels like Apple does not have a real plan now and are just trying to catch up.

Yes – or a use case. Apple seems very happy to advertise that they keep their inference on-device.

There’s no way to do freeform chat on device. On-device models can realistically be used for constrained tasks like summarization, RAG, email/instant message replies, that kind of thing. You aren’t going to get anything GPT4 like unless your “device” can fit a model with at least tens, if not hundreds of billions of parameters. Which, to be fair, a number of higher end Apple SKUs can.

I think Apple’s said before that they don’t really get the chat usecase, their priority is integrating ML backed features into the OS (text highlighting in images, etc).

It seems that even the state of the art models can’t be used to make a more reliable Siri without another few breakthroughs, so it’s probable that these new AI features aren’t chat stuff.

Oh, I think we’ve passed that point already. Llama 3 8B works incredibly well for chat. On the LMSYS English-only leaderboard, it’s only 3 points below GPT-4-0314. (!)

It will hallucinate, as every other LLM will also do, so it shouldn’t be used as a store of knowledge. But grounded with context, it is astonishing in its capability.

For that matter, Phi-3 at half the size is not bad at all.

Yeah, if you’re OK with it making everything up, it “works incredibly well”. I’m not ok with that, so I actually prefer models where hallucination is minimized as much as possible because they encode massive amounts of knowledge. Points don’t matter. Of all SOTA models, GPT4, the largest one, hallucinates the least. Almost not at all, in fact.

It doesn’t “make everything up”.

On a hallucination benchmark [1] Llama 3 8B has a hallucination rate of 5.4%, compared to GPT4 at 3%, and GPT-4 Turbo at 2.5%. In absolute terms these just aren’t huge differences.

[1] https://huggingface.co/spaces/vectara/Hallucination-evaluati...

Benchmarks don’t tell the whole story. Obscure questions are unlikely to be in benchmarks. Large models simply encode more knowledge, which means that they are _way_ more likely to not hallucinate an answer to practical questions that are off the beaten path and that the model hasn’t seen before. Try asking your Llama 8B about some less commonly known details in some topic you know well, and then repeat the same with GPT4. Even Claude Opus hallucinates way more than GPT4, despite the claims that it matches it according to the various leaderboards and whatever.

I agree that larger models have more memorization capacity, but that isn't what find exciting about LLMs, personally. You can ask GPT-4 some esoteric question and get back an answer that's more likely to be truthful than Llama 3's, but either way, it's the same result: you have to look it up for yourself.

What I think is much more interesting about LLMs is their ability to transform natural language inputs. Applications like open-book question answering, summarization, data extraction, and so forth. For these things, small models are at the point where they are commercially useful now. And if you do fine-tune them on your own domain, they will work better than larger general-purpose models.

Bit strange. I thought they don’t even do their own storage (at least partially on GCP)

Maybe it’s supposed to be complementary to their on device strategy. Ie split tasks by difficulty

Network observations have shown Apple traffic going in and out of AWS and GCP. But they also own their own data centers, for example:

https://www.apple.com/newsroom/2017/08/apples-next-us-data-c...

I feel like we screwed up the AI chip segmentation with separate “inference” and “training” chips

Every machine learning application ever involves both training and inference.

What is the point of a model that can’t learn?

What is the point of learning a great model that is never used?

It seems like these lead to different optimizations, and they do.

But the bigger lock-in is the SW stack.

If you own one side of it, you have a huge advantage in SW compatibility that you can leverage into the other.

Most inference can be easily abstracted over multiple ISV implementations so the inference side rarely is locked in. It’s usually the training side or software above the inference layer that’s locked in.

But as to your question as to why: silicon size and frequency of use.

You’re almost always going to have a high ratio of inference to training.

Silicon for training needs to be much more capable due to how intensive it is. That’s then a block of silicon that isn’t used to its full potential, takes up valuable space and in turn jacks up cost to buy and cost to run (energy).

It’s simply not a worthwhile trade off.

The point of the separation is economic, owners create and train models, and renters are charged to execute them on their edge machines.

> Every machine learning application ever involves both training and inference.

Not really, no. There are a lot of "classical" models that you don't want to update on the fly. I'd even argue that it's a minority.

https://www.msn.com/en-us/money/other/apple-is-developing-ai...

I think this makes sense for Apple to also provide the cloud capabilities similar to NVIDIA DGX so that any AI models that are targeted to Apple chips can run scaled up in data centers but Apple owns the front end for it as well as any ongoing optimizations

Most of the profits in AI chips for data centers will be in selling and renting said chips.

Is Apple going to start offering a cloud or selling chips directly? That would be very unexpected

Of all the big tech companies, Apple seems to have missed the boat more than others

they're going to make the chips, and build a cloud with them and then they're going to sell services with that cloud directly to the consumer.

they're going to own the entire stack, from fab to finger

Apple doesn't own a fab currently afaik, that would be new as well

Apple is in no position to own fab and besides, we’d hear about it many months prior. They aren’t going to be able to keep that secret.

My guess is that they’d use TSMC or perhaps even Intel, for fab. But probably TSMC.

Apple has more cash on hand than nearly every private sector entity on Earth. They could build a fab with zero investment or credit activity.

Yet they are choosing to do a $110B stock buyback, which apparently exceeds their cash on hand in the books

> Since Apple’s $110 billion stock buyback exceeds its balance sheet cash, how will the iPhone maker pay for the share buyback?

https://www.forbes.com/sites/petercohan/2024/05/05/warren-bu...

Buybacks happen over a period. Cash on hand isn’t the only factor, cash flow in is a large one.

Apple stated several years ago that they’re aiming to be cash neutral

https://www.fool.com/investing/2018/02/02/understanding-appl...

In the earnings call last week they stated they’re still a while away from that goal. These buybacks are part of the net cash neutral strategy.

Other forms they could take are upping R&D (but they already seem to be pretty high), or increasing employee wages (that would be much appreciated I’m sure) or reducing product margins (also probably appreciated by a different group). Though the latter two are harder to back out of if things ever go south.

Can you ELI5 for someone who doesn't know corporate finance why constant cash surpluses would be unfavorable for them?

Honestly, I do not understand it myself either. I just understand the mechanisms behind it but I do not understand why one would logically want less money.

If you owned a business that had a bunch of cash sitting around with no useful growth opportunities to invest it in you wouldn't keep it in your business's bank account, you'd pay yourself and spend the money on a vacation or invest it elsewhere.

Same with a giant corporation. Its shareholders want to be invested in a business. Owning part of a business that is 10% just a giant cash pile isn't necessarily appealing. The corporation should return the cash to its investors so they can decide what they want to do with it.

That’s fair. I’d probably invest it in my employees though personally rather than return to investors, but I follow.

Sitting on cash means you aren't putting it to use.

1. invest in the business to increase long-term profits

2. give out dividends to investors

3. buy back stocks so future dividends are more valuable

Cash flow (from revenues) is relatively predictable, so you can count on more coming in to cover future expenses. Business can also issue bonds (loans) should cash be in shortfall or they want to invest in a factory or similar.

Even if they could, they haven’t.

And if they did, it would take months/years and we would hear about it.

Right after buying Disney, too.

Buying Disney?

No doubt Apple can negotiate some form of super-exclusivity deal with a fab, in the same way that their license from ARM is probably more permissive than any other of their partners.

(I wouldn't put it past them to go with Intel – after all, during the PPC era, it also would've been unexpected to go with Intel!)

They’ve gotten exclusive access to nodes by buying up the full capacity before.

I doubt Apple would sell the chips, or even rent them. And they likely won’t even make the models directly available, even as a cloud service, they’ll be embedded in products like Xcode.

I think that’s how they’d rent them. You’d set up a training project in Xcode and press a button and as part of your Xcode cloud subscription it would run on Apple’s hardware instead of locally.

Maybe you are billed by time used. Maybe you have to buy hours upfront. I could see either.

They already have Xcode Cloud. Perhaps you could run your training jobs there and they would run on Apple’s new custom silicon. You’d pay for it as a part of your existing subscription.

That + internal use would make sense to me.

Reply on news.ycombinator.com