Hacker News Reader

fsbonettoon OpenTPU – An open-source AI accelerator, developed by AI

Cool!

fsbonettoon OpenTPU – An open-source AI accelerator, developed by AI

Yup, consider contributing to te project ;)

fsbonettoon OpenTPU – An open-source AI accelerator, developed by AI

Yeah thats true, chip to chip communication and KV cache management is make or break for an AI ASIC.

fsbonettoon OpenTPU – An open-source AI accelerator, developed by AI

It's not CPU based... What do you mean?

fsbonettoon OpenTPU – An open-source AI accelerator, developed by AI

It's at 80~90% the max perf it can achieve on this hardware... Which is DDR3 speeds. I'm working on getting FPGA's with DDR4 and HBM2 next.

fsbonettoon OpenTPU – An open-source AI accelerator, developed by AI

Yup, the tests already compare to a CPU hugginsface implementation, but I'm working on making running and verifying the results of those tests easier and more available. Also, working on fixing those bugs and make the code more reliable now that the underling hardware has been saturated (memory bound)

fsbonettoon OpenTPU – An open-source AI accelerator, developed by AI

Indeed there are some bugs/"non standard behavior" regarding very small or very big floating point values. All of those where proven harmless for LLM inference. Thanks for pointing out the bug. If you caught something outside of that, please point that out so I can fix it ;)

fsbonettoon AI is now capable of developing its own inference hardware

First opensource arch that can do modern LLMs, while maximizing the potential of its hardware; First opensource TPU build by a recursive improvement loop...

It's upcoming second generation could run the inference of the models that are being used to improve it...

fsbonettoon AI is now capable of developing its own inference hardware

It could have a small improvement on power consumption, but the current design can already achieve 90% of the maximum theoretical speed of this hardware without giving up programability/flexibility

fsbonettoon OpenTPU – An open-source AI accelerator, developed by AI

They are more like a way to proving the architecture of the accelerator before committing 100's of millions into a custom ASIC with TSMC

fsbonettoon OpenTPU – An open-source AI accelerator, developed by AI

GPUs are faster, but you can't make your own arch on GPUs. FPGAs offer you that possibility. Said that... There are a few beasty FPGAs used in crypto mining coming my way... I expect that OpenTPU will be able to run frontier models with those.

fsbonettoon OpenTPU – An open-source AI accelerator, developed by AI

The bottleneck, for inference at least, is memory bandwidth. And that you can't make any faster by making it specific to your model.

So companies try to maximize the memory bandwidth they can get, balancing tradeoffs of power/area/programability of their chip. Right now they feel like the economy on power/area is not worth the decrease in programability/flexibility.

fsbonettoon AI is now capable of developing its own inference hardware

Its a datacenter decommissioned board, really popular among hobbyists.

For a TPU focused on inference the name of the game is memory bandwidth. How much of the available bandwidth you can extract for as little logic/area/power as you can.

fsbonettoon AI is now capable of developing its own inference hardware

Besides an specific movie ? There is this CPU auto improving loop as well: https://github.com/FeSens/auto-arch-tournament Opus 5.5 was the first to beat the human baseline

fsbonettoon OpenTPU – An open-source AI accelerator, developed by AI

After using AI to develop risc-v CPU cores, the same technique was used for developing openTPU. An open source AI inference engine. It's able to run most of the modern models like Qwen 3.5, Gemma 4, and many others. The TPU started able to produce only a few tokens per second and trough a recursive self improvement loop got to 80+ tok/sec on the smallers models.