Hacker News Reader

Comment by Workaccount2 | Hacker News Reader

I'm continually impressed how Google manages to be left out of LLM discussions. I don't know what it is but it always seems like people forget about Google in this space.

Google has both cutting edge SOTA AI models and has it's own in-house ai acceleration hardware that is approximately on par with Nvidia. It's why their models are dirt cheap to use and have enormous context.

View on HN

Until you can buy a TPU and stick it in your own server, Google may as well be just another commodotized provider on OpenRouter.

They're more or less dev boards, but they absolutely do sell TPU modules that you can stick right into a M.2 or mini PCIe slot.

https://coral.ai/products/

I have to say these seem like hobbyist-level products. For example https://coral.ai/products/m2-accelerator-dual-edgetpu can do 8 TOPS, but a 5-year-old RTX 2060 gets you 50 TOPS. A newer H100 gets you 3958 TOPS.

Nobody's going to buy 500 of those chips and stick them in 500 M.2 slots to match the performance of a single H100.

Do they make any better chips that I missed?

I would call them less hobbyist products, and more compute for IOT/edge devices. They aren't made for a datacenter and aren't trying to compete with an H100.

Yes, it has one sixth the performance of a RTX 2060, but it has one-five-hundredth the volume. For a specific siloed application, 8 TOPS is plenty. Think image processing, etc.

There are plenty of production use cases where that makes sense and an H100 does not.

At this point, these chips aren't anywhere close to the frontier of capability for embedded accelerators, and certainly don't merit an entire M.2 slot in a serious design.

These are at 5-year-old design and it shows.

Fair, the actual M2 version linked would not be what would be used beyond convenient development.

The actual TPU chip itself from the M2 card is sold individually as a surface mount chip and that is what would be used in a "serious" design.

https://coral.ai/products/accelerator-module

I did a cursory search and I can find zero other products that compete with that module, within an order of magnitude of power consumption/price/etc

At this point, nobody sells modules for this, and I doubt many coral chips still sell. The current slot for an ML accelerator at about 10 TOPS is as a peripheral on an SoC. Most serious SoCs have one.

In other words, the reason you didn't find a commercial competitor to these things is that the competitor is (nearly) free.

Here's one vendor: https://www.ti.com/technologies/edge-ai.html

I didn't know this form exists - sounds ideal for a project I had in mind. Thanks for posting!

If the company owns both frontier models and chip design and they see the future moat is in inference why would they offer much more than what you get on Google Cloud? Is not as if they're gonna start competing with Nvidia in hardware anyway, this is a very specific hardware design for a very specific problem.

All of this is true, but it's somewhat offset by the possibility that LLMs may disrupt Google's core revenue stream from search.

Reply on news.ycombinator.com