gchadwick

I'd disagree as the K, Q and V have distinct functions within the attention calculation. In particular when you're considering decode (next token calculation during inference which follows the initial prefill stage that processes the prompt). For decode you have a single Q vector (relating to the in progress token) and multiple K and V vectors (your context, i.e. all tokens that have already been computed).

> you can represent all of the transformer as wide single layer perceptron sequences

This isn't correct, again because of attention. The classic perceptron has static weights, they are not an input. The same mathematical function can be used to compute attention however there are no static weights. You've got your attention scores on one side and the V matrix on the other side.

Indeed I wonder if it's actually possible for a bunch of perceptrons to even 'discover' the attention mechanism given they inherently have static weights and they can't directly multiply two inputs (or directly multiply two internal activations). Given an MLP is a general function approximater I guess a sufficiently large number of them could get close enough?

A real issue here is lack of training data (at least for LLMs). There's lots of high quality (and plenty more poor quality) open source software that can be used to train on. There's significantly less open source hardware and often the stuff that does exist is mostly front end design. Good examples of complete test benches (ones you'd close verification on and go to a production tape out with) are few and far between and there's basically nothing for modern physical design and backend considerations (i.e. how you take your design and actually manufacture a chip with it).

Commercial companies who may be interested in AI tools for EDA do have these things of course but are any going through the expensive process of fine tuning LLMs with them?

Indeed perhaps it's important to include a high quality corpus in pre training? I doubt anyone wants to train an LLM from scratch for EDA.

Perhaps NVidia are doing experiments here? They've got the unique combination of access to a decent corpus, cheaper training costs and in house know how.

This whole series is fantastic! Does an excellent job of explaining the theoretical limits to running modern AI workloads and explains the architecture and techniques (in particular methods of parallelism) you can use.

Yes it's all TPU focussed (other than this most recent part) but a lot of what it discusses are generally principles you can apply elsewhere (or easy enough to see how you could generalise them).

> It's really hard to describe how bad the general vibe is here.

Eh, think that depends upon your social circles, sure it's not perfect but the vibes just fine from my perspective. There seems to be a massive swelling of online opinion that everything's terrible and everyone's deeply unmotivated which certainly doesn't match lived reality for me

This certainly rings true with my own experiences (worked on both GPUs and CPUs and now doing AI inference at UK startup Fractile).

> If there were a hypothetical high-level CPU language that somehow encoded all the information the microarchitecture needs to measure to manage memory access,

I think this is fundamentally impossible because of dynamic behaviours. It's very tempting to assume that if you can be clever enough you can work this all out ahead of time, encode what you need in some static information and the whole computation just runs like clockwork on the hardware, no or little area spent on scheduling, stalling, buffering etc. Though I think history has shown over and over this just doesn't work (for more general computation at least, more feasible in restricted domains). There's always lots of fiddly details in real workloads that surprise you and if you've got an inflexible system you've got no give and you end up 'stopping the world' (or some significant part of it) to deal with it killing your performance.

Notably running transformer models feels like one of those restrictive domains you could do this well in but dig in and there's plenty enough dynamic behaviour in there that you can't escape the problems they cause.

Yeah it's a bit odd. I can get a sense of what they might be doing and perhaps it's a compelling product but no where near enough info to go on, let alone reserve one for $99 and later buy at $1999.

I'd argue the last one looks better as well, at least if you're considering what looks more 'real'. The ChatGPT one looks like it could have been a shot from a film, the Krea one looks like a photo someone took off their phone of a person heading into a car park on their way back from a party dressed as a super hero (which I think far better fits the vibe of the original image).

Immobilizers (which lock out the engine until there's been some authorization from another device, i.e. from a chip in the key) have been mandatory in cars in the UK at least (and I would presume Europe on similar time scales) for almost 30 years (from 1998). Seems they've been sold in cars for a few years longer than that (from 1992). According to: https://www.carwow.co.uk/guides/glossary/what-is-a-car-immob....

Maybe never introduced into the US market? Would find that hard to believe.

> The interconnect between tiles is also statically routed and bufferless, decided at compile time. As there's no flow control or retry logic, if two data paths would normally collide, the compiler has to resolve it at compile time.

This sounds like the most troublesome part of the design to me. It's very hard to do this static scheduling well. You can end having to hold up everything waiting for some tiny thing to complete so you can proceed forward in lock step. You'll also have situations where 95% of the time the static scheduling can work but 5% of cases where something fiddly happens. Without any ability for dynamic behaviour and data movement small corner cases dominate how the rest of the system behaves.

Interestingly you see this very problem in hardware design! All paths between logic gates need to be some maximum length to reach a target clock frequency. Often you get long fiddly paths relating to corner cases in behaviour that require significant manual effort to resolve and achieve timing closure.

Both? Theoretically amazing hardware that just needs the magic compiler to work well is a well worn path in the hardware world (The itanium being a notable example). A design can be impossible to compile well for and very hard to program manually if it hasn't been developed well. Equally you can indeed have a bad toolchain for that hard to use design making it even harder to get the best out of it.

An excellent example I really should have been able to think of myself!

I do wonder why they decided to have have separate shell size and pin designations given there appears to be a 1:1 correlation between shell sizes and pins (i.e. the 'B' shell is always 25 pins, the 'E' shell is always 9 pins). Perhaps there was plan to have fewer pins in the same shell at some point?

The real bottleneck is the memory, optimize your matmul architecture all you like whilst you still have it connected to a big chunk of HBM memory (or whatever your chosen high bandwidth memory is) you can only do so much.

So really GPU v not GPU (e.g. TPU) doesn't matter a whole lot if you've got fundamentally the same memory architecture.

Very glad to see this. There is (understandably) much excitement and focus on training models in publicly available material.

Running them well is very important too. As we get to grips with everything models can do and look to deploy them widely knowledge of how to best run them becomes ever more important.

> It has two layers

Where did you get that from? Pretty sure it's a single planar set of transistors. Those transistors are manufactured using multiple layers of mask.

FinFET transistors are described as 3D or non-planar but crucially this isn't allowing transistor on transistor stacking you've just got the gate structure of the FinFET poking out above the plane of the rest of the transistors.

Silicon on silicon die stacking is a possibility but limits your power and GPUs run very hot so it's not an option for them.

The wires sit on top of the transistors. Many layers of them in a modern process.

However you can't always pack the transistors as dense as you would like because you can't fit the wiring for them in above at the same density.

Plus there are various 'design rules' that constrain how things get placed. These are needed to ensure manufacturing is successful and achieved good yield. An important set of rules are the 'antenna rules' that requires the insertion of antenna diodes (using silicon reducing transistor density) to prevent circuitry being destroyed during manufacturing: https://www.zerotoasiccourse.com/terminology/antenna-report/

I wonder if they ever taped this out and did some device characterization?

The devil really is in the details with things like this. 1 in 10 million events which may be hard to trigger in a detailed analogue simulation can occur many times per second when you run it in silicon and make it effectively useless.

Still very cool to have enough open source tooling and a PDK so this can even be attempted as a piece of open source design work!

It's even worse when paired with the ability for a local authority to go bankrupt when it can't cover the bills and be forced to sell off major capital assets (e.g. buildings, sometimes of significant public interest like concerts halls, leisure centres and other community venues).

Of course the actual place continues to exist so the local authority will continue to exist in another form, this time with fewer major capital assets and they're paying rents to the people who now own them instead.

As pointed out in the article you could see this happen when something entirely out of the authorities control (e.g. spending on SEND children due to the massive increase in eligible children in Central Bedfordshire's case) causes it too.

I think a key part of the 'poor' feeling in day to day experience comes from councils' inability to do maintenance, things like pot holes, children's play equipment, public toilet, general upkeep on public spaces and services like libraries. In the grand scheme of things this isn't too expensive but it's been cut to the bone due to way local government funding works. This is explored in the article:

> in large part because they’re mandated to write blank cheques for social care with no support or strategy from central government. Individual cases in Central Bedfordshire are now costing up to £750,000 per year, a quarter of the entire libraries and leisure budget and an amount that is rising rapidly with no apparent ceiling. As I wrote previously, “In a single year, residential care costs for children have increased by £2,000 per child… per week,” taking the average cost for a single case from ~£200,000 to ~£300,000 per child per year, again with little explanation as to where the money is going or how this is even possible.

> Similarly, “school transport costs have increased by over 100% - from £9m to £20m - in just 4 years” - that’s driven by an unexplained rise in the number of SEND pupils eligible for support and it amounts to roughly the same as - deep breath - the transport, roads, parking, libraries, leisure, housing benefit, public protection and safety budgets combined. Central Bedfordshire Council is not an outlier here - collectively, council overspends on SEND services are set to hit £2bn in the next year, risking further bankruptcies. Again this is not about pitting children against libraries, but asking if we seriously believe we’re addressing either of these things well?

Local councils have to pay the very large bills for social care and supporting SEND children but have basically little control over how it's spent or levers to help control the bills.

Fixing this so councils can once again spend relative minor amounts of money improving the public realm could go a long way to improving day to day experience. Definitely some other large structural problems (see the huge costs of HS2) but it would provide a noticeable improvement in people's lives and potentially isn't too hard for a government willing to make some bold changes around taxation, local government funding and providing proper national strategy and funding on social care.

> ChatGPT can just send you something that is completely wrong, and you have no way of knowing.

This is true, if you decide to take a ChatGPT answer at face value without any further work. Personally I find it useful sometimes to ask an LLM a question, get an answer and the verify that answer for myself. Doing web searches and pulling together relevant information to get the answer for a question can be harder than getting an answer and then looking to verify it. Perhaps something like that was going on here, impossible to know of course.

A dupe of my comment from another post on this (relating to the US arbiter ruling that she may not promote the book https://news.ycombinator.com/item?id=43351949)

People may be interested in the interview with Wynn-Williams (the whistleblower) on the News Agents podcast: https://www.globalplayer.com/podcasts/episodes/7DrpKCA/ (it's a UK news/political podcast very popular in the UK). From what they said at the beginning I think this is her first big podcast interview about the book/her claims. I wonder if she chose a UK podcast because of the US arbiter ruling.

People may be interested in the interview with Wynn-Williams (the whistleblower) on the News Agents podcast: https://www.globalplayer.com/podcasts/episodes/7DrpKCA/ (it's a UK news/political podcast very popular in the UK).

From what they said at the beginning I think this is her first big podcast interview about the book/her claims. I wonder if she chose a UK podcast because of the US arbiter ruling.

Very sad to see Efabless go. They really did offer something unique with their chipIgnite programme. It provided a simple way to get a chip made at very low cost ($10k) with no barriers. Thanks to the open source toolchain and PDK you just needed your verilog and a credit card and you were good to go. Sure it had a bunch of limitations but it was still a very encouraging development in making chip design more accessible.

Other low-cost (at least low relative to the huge costs of silicon fabrication) services exist but typically they have little public information and will certainly require proprietary tools and PDKs (which are not cheap and require you to persuade the sales people to talk to you to even find out what these costs are).

Personally I find value in toy examples like this as it helps me check I actually understand the core architecture. You can look at a diagram or an equation explaining a transformer block but how do you make the jump from that to actually implementing it? With such examples you can check your understanding of what's actually getting computed where, what the dimensions of the vectors and matrices are etc.

Sure the vast majority of people who want to work with LLMs don't need to know any of this. But personally I enjoy understanding how things work and I like to be able to drill down into what's happening at a very basic level. Something like this is a lot easier to play around with and check your understanding vs the source code of a real transformer model. Once you feel you've mastered a toy example like this, the real code becomes easier to get to grips with.

Worth noting that production silicon is on the way for OpenTitan (https://opensource.googleblog.com/2025/02/fabrication-begins...). An OpenTitan chip in a future TKey can give both strong hardware security (you'll have real silicon resistant against hardware attacks, not just an FPGA) whilst remaining full open source hardware (you can see all the RTL here: https://github.com/lowRISC/opentitan/tree/earlgrey_1.0.0 available under an Apache 2.0 license).

Only given it a skim read so far but this looks like a really nice write up and it's a fantastic piece of work. Laser fault injection is a very powerful attack so great to see it becoming more accessible through work like this (or terrible if you're someone relying on security of hardware that considers LFI to be a highly advanced technique that it doesn't need to defend against...).

I do find HN threads about R-Pi products weigh heavily negative. Yet despite the apparent many major issues that keep getting pointed out R-Pi seems to keep going from strength to strength. Maybe they understand their own customers and their market better than commenters here?

Thinking about it the Apple II is a better example here, for one thing it's yet older (47 years). However I don't personally posses one nor have I ever used one so I concentrated on the C64.

The author talks about doing timeless activities well. You can still word process, do spreadsheets and program on an Apple II. Probably meets the author's 'sturdy and resilient' requirements as well as being a 'heavier and well-designed object'.

I find it odd the author spends lots of time talking about vintage typewriters but then fails to consider vintage computers which can give some real life examples of computers that are still usable almost 50 years on from their original release. E.g. the Commodore 64, lots of working examples still around and now 42 years since first release.

Certainly a C64 is highly restrictive compared to a modern machine and were one to specifically build a computer to last 50 years it's not where you'd start but surely a machine that has actually lasted almost 50 years and remains usable has things to teach you about long lasting computer design.

In particular interesting to see how open source fits in. The modern C64 ecosystem has plenty of tools and utilities that do use open source software and hardware (e.g. the Kung Fu Flash cartridge: https://github.com/KimJorgensen/KungFuFlash) but plenty of the core software, that actually runs on the machine, is proprietary software the source is long gone for. It's still around because of archivists and pirates and can continue to be used because the original copyright holders don't care to enforce their copyrights. So is open source actually a core item as the author asserts or just a nice to have? Having the software be archived and easily available later was the key. Along with simplicity, you just run the monolithic binary, there's no dependencies and the software is sufficiently simple that hacking around with the raw binary is perfectly feasible.

> After "bootle cap attached to the bottle" EU law

Am I the only person who thinks this is just fine and indeed a good idea? Minor tweak to lid manufacturing that aids recycling and disposal and is useful as a user of the bottle by ensuring the lid doesn't get lost. Though going by internet commentary it's apparently a huge imposition that totally ruins bottled drinks. May well be a silent majority thing? If many like me looked at it once and went 'huh interesting change, seems reasonable' they're not jumping onto to twitter to talk about it.

As for the subject of the article it's just an idea from a Dutch PhD student hardly a serious EU proposal.