zachbee

The interesting thing here isn't "how, logistically, is the Groq corporate entity able to raise more money?". That's straightforward.

Rather, the interesting thing and the topic of most of the article is "how, after Nvidia hired most of Groq's team and licensed all their IP, did Groq manage to convince investors to invest in the remaining corporate entity?"

Their AES implementation uses old-school 2-share boolean masking [1], which has been shown to be insecure since 2005 [2][3]. A modern implementation would use domain-oriented masking [4], like OpenTitan does. Pretty bad look for Crossbar.

[1] https://github.com/baochip/baochip-1x/blob/main/rtl/modules/... [2] https://link.springer.com/chapter/10.1007/978-3-540-30574-3_... [3] https://static.aminer.org/pdf/PDF/000/086/973/successfully_a... [4] https://eprint.iacr.org/2016/486

Their AES implementation uses old-school 2-share boolean masking [1], which has been shown to be insecure since 2005 [2][3]. A modern implementation would use domain-oriented masking [4], like OpenTitan does. Pretty bad look for Crossbar.

[1] https://github.com/baochip/baochip-1x/blob/main/rtl/modules/... [2] https://link.springer.com/chapter/10.1007/978-3-540-30574-3_... [3] https://static.aminer.org/pdf/PDF/000/086/973/successfully_a... [4] https://eprint.iacr.org/2016/486

One thing that people rarely discuss about book publishing is a change to US tax law in the late 70s that meant that publishers couldn't write down the value of unsold inventory, but could write off that inventory by destroying it.

That meant that poorly selling books were destroyed to realize a taxable loss, which killed the ability for books to slowly "pick up steam" over a year or two to eventually generate a profit for the publisher. If you didn't make a profit fast, the backlog got destroyed and the book lost its chance to make money.

It's a cute toy and a fun educational tool, but "computing for the future" seems like a bit of an overstatement.

hi, this is my article! thanks so much for the views, upvotes, and comments! :)

Author here -- I don't disagree! I actually noted this in the article:

> Well, it turns out that LLMs are also pretty valuable when it comes to chips for lucrative markets -- but they won’t be doing most of the design work. LLM copilots for Verilog are, at best, mediocre. But leveraging an LLM to write small snippets of simple code can still save engineers time, and ultimately save their employers money.

I think designers getting 2x faster is probably optimistic, but I also could be wrong about that! Most of my chip design experience has been at smaller companies, with good documentation, where I've been focused on datapath architecture & design, so maybe I'm underestimating how much boilerplate the average engineer deals with.

Regardless, I don't think LLMs will be designing high-performance datapath or networking Verilog anytime soon.

I didn't get into this in the article, but one of the major challenges with achieving superhuman performance on Verilog is the lack of high-quality training data. Most professional-quality Verilog is closed source, so LLMs are generally much worse at writing Verilog than, say, Python. And even still, LLMs are pretty bad at Python!

Yes, it would be a paid role. But I also don't necessarily want my job to be in jeopardy if my boss has a bad breakup.

If they're doing inference on edge devices, one challenge I see is protecting model weights. If you want to deploy a proprietary model on an edge AI chip, the weights can get stolen via side-channel attacks [1]. Obviously this isn't a concern for open models, but I doubt Apple would go the open models route.

[1] https://spectrum.ieee.org/how-prevent-ai-power-usage-secrets

I'm curious what the motivation is here -- unfortunately, the dev blog is all in Chinese and I can't read it. If it's mostly to show a proof-of-concept of LLMs on a FPGA, that's awesome!

But if this is targeting real-world applications, I'd have concerns about price-to-performance. High-level synthesis tools often result in fairly poor performance compared to writing Verilog or SystemVerilog. Also, AI-focused SoCs like the Nvidia Jetson usually offer better price-to-performance and performance-per-watt than FPGA systems like the KV260.

Potentially focusing on specialized transformer architectures with high sparsity or significant quantization could give FPGAs an advantage over AI chips, though.

Not to toot my own horn, but I wrote up a piece on open-source FPGA development recently going a bit deeper into some of these insights, and why AI might not be the best use-case for open-source FPGA applications: https://www.zach.be/p/how-to-build-a-commercial-open-source

I'm definitely not! I'm a hardware designer and I work with FPGAs all the time, for both work and for personal projects. Like with all things, there's a right tool for every job, and I think for modern DL algorithms like Transformers, GPUs and AI ASICs are the better tools. For rapid hard prototyping, or for implementing specialized architectures, FPGAs are far better.

The big challenge when it comes to using FPGAs for deep learning is pretty simple: all of that reprogrammability comes at a performance cost. If you're doing something highly specific that conventional GPUs are bad at, like genomics research [1] or high-frequency trading [2], the performance tradeoff is worth it. But for deep learning, GPUs and AI ASICs are highly optimized for most of these computations, and an FPGA won't offer huge performance increases.

The main advantage FPGAs offer is being able to take advantage of new model optimizations much earlier than ASIC implementations could. Those proposed ternary LLMs could potentially run much faster on FPGAs, because the hardware could be optimized for exclusively ternary ops. [3]

Not to toot my own horn, but I wrote up a blog post recently about building practical FPGA acceleration and which applications are best suited for it: https://www.zach.be/p/how-to-build-a-commercial-open-source

[1] https://aws.amazon.com/solutions/case-studies/munich-leukemi...

[2] https://careers.imc.com/us/en/blogarticle/how-are-fpgas-used...

[3] https://arxiv.org/abs/2402.17764

If I was developing an AI app, I'd care about quality first before speed. And the open-source models just aren't as good as the closed ones.

Totally saw this one coming! [1]

I think one major challenge they'll face is that their architecture is incredibly fast at running the ~10-100B parameter open-source models, but starts hitting scaling issues with state-of-the-art models. They need 10k+ chips for a GPT-4-class model, but their optical interconnect only supports a few hundred chips.

[1] https://www.zach.be/p/why-is-everybody-talking-about-groq

This research is awesome. And I think organic semiconductors are super promising, especially looking at the success of OLED.

But the IEEE article is missing the biggest pitfall of this tech right now: it only works well at very low temperatures (80K and lower).

"Smart home" has really fallen flat across the board, from smart speakers to the Roomba. It would be great if Apple changed that, but I'm not optimistic. I just don't think people really want smart home devices all that much.

This doesn't appear to support any FPGAs other than their FPGA chiplet [1]. Also, like UncleOxidant said, the most complex parts of this toolchain are just existing open-source tools (Yosys and VPR).

This seems like a useful toolchain for ZeroASIC customers who are using their hardware, but not for FPGA enthusiasts more broadly.

[1]. https://www.zeroasic.com/chiplets/fpga

There are meaningful size and performance challenges to implementing post quantum crypto, but I don't think "oh no, HTTPS handshakes are 20% slower!" is something we should actually worry about.

The real worry is all of the small, secure, embedded devices that literally don't have enough memory or compute to run these algorithms at all. Even the state-of-the-art hardware-accelerated implementations of PQC use a ton of memory and a ton of silicon area, to the point where they're untenable for lower end devices.

I'm really skeptical of Eteched. They make claims about how their chips are "optimized for transformers", as if Nvidia's Tensor Cores aren't optimized for those workloads too. Plus, if they really are only good at transformers, they'd get screwed over if SSMs or some other models become dominant. And their website has a "tokens per second" graph with no Y-axis, which is never a good look.

I think that if you want to build an AI chip company, you have to leverage some novel technology (processing-in-memory, photonics, etc) that counter-positions you against Nvidia. Trying to beat Nvidia at optimizing digital logic seems like a losing battle.

I'm a hardware security engineer, and I've actually used similar tools before! It's nifty, but overall, hardware security tooling is unfortunately seriously lacking.

When you try to apply masking or hiding schemes to prevent side-channel leakage of logic (see [1]), the design tools will often optimize your designs away and make your logic insecure! The same goes for certain parity and polarity propagation schemes used to protect data on busses from snooping attacks.

[1] https://www.iacr.org/archive/ches2014/87310197/87310197.pdf

From a technical perspective, moving a mouse with your mind isn't entirely new: https://elifesciences.org/articles/18554

Personally, I think the most exciting part of Neuralink and other companies working on BCIs is the fact that they're trying to keep these implants in long-term, and scale the deployment significantly. Most academic BCI research thus far has just been trials, without patients getting to keep the implants long term.

There are so many different tools to help design and route PCBs now. You have DeepPCB, Flux, and Quilter doing auto-routing, and SKiDL, CuFlow, and JITX doing programmatic schematics (and also sometimes auto-routing?).

To be honest, I don't really understand how these tools are differentiated from one another. I personally know the Quilter guys, so if I had to pick I'd use their product, but I don't think the market crowding here is great for anybody.

When they originally announced tiny corp and the tinybox, the entire pitch was that AMD hardware were great, but their software was bad. [1] Now they're giving up on AMD hardware because the software is bad. Wasn't the whole point to solve that problem?

I'm hopeful for tinygrad as a piece of software, but I'm skeptical about the future of the tinybox if they keep waffling on the hardware so much.

[1] https://geohot.github.io/blog/jekyll/update/2023/05/24/the-t...

100% right. GF is no longer gunning for leading-edge nodes. They've made their niche in mature nodes for low-power chips and emerging memories (FeFETS, MRAM, RRAM, etc). If you want to get leading edge nodes on American soil, Intel is the best bet.

I dunno man, I think saying "this is a personal project of mine" and linking to an open source github repo implies that it's a casual side project. I never made any performance claims, it's just for fun. I'm not trying to put up clickbait.

And yeah, I think the fact that you can build a classical, analog, D-Wave-sans-qubits machine on an FPGA is cool!

In practice, though, asynchronous systems can run a lot faster than their synchronous counterparts! Also, I'm mostly working on this project because it's cool and interesting to build oscillator-based computing systems on FPGAs, not because I'm trying to build a practical system.

That's why I posted it here: it's a fun use-case for asynchronous stuff you can do on FPGAs.

The Fujitsu device is a synchronous digital architecture. I'm using asynchronous oscillators, more like Lo et. al. (https://www.nature.com/articles/s41593-024-01607-5)

I personally think it's super cool that we can replicate analog coupling dynamics using asynchronous digital logic!

This is a personal project aiming to replicate analog coupling dynamics with asynchronous digital circuits FPGAs, not a claim that we can solve NP complete problems with a fundamental speedup.

Also, these sorts of Ising architectures get their speed-up are based on oscillator interaction, not parallelism. It's relevant to this article because it leverages oscillators on FPGAs in a nifty way :)

I've actually been working on a personal project using asynchronous ring oscillators on FPGAs to perform meaningful computation. If you make the oscillators interact in the right way, you can leverage them to solve certain graph problems!

Once I finish the write-up I'm planning on posting it on HN :)

https://github.com/zbelateche/digial-ising