fooblaster

I wouldn't trust any benchmarks on the vendors site. Microsoft went down this path for years with FPGAs and wrote off the entire effort.

Yep, but they are still 50x faster than any fpga.

Show me a single FPGA that can outperform a B200 at matrix multiplication (or even come close) at any usable precision.

B200 can do 10 peta ops at fp8, theoretically.

I do agree memory bandwidth is also a problem for most FPGA setups, but xilinx ships HBM with some skus and they are not competitive at inference as far as I know.

Yeah, I wouldn't have guessed it would be helping me write systemverilog.

FPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore. 50% of the die area or more is for fixed function matrix multiplication units and associated dedicated storage. This just isn't general purpose anymore. FPGAs cannot rival this with their configurable DSP slices. They would need dedicated systolic blocks, which they aren't getting. The closest thing is the versal ML tiles, and those are entire peoxessors, not FPGA blocks. Those have failed by being impossible to program.

Great! How do you program it?

Well, anthropic just purchased a million TPUs from Google because even with a healthy margin from Google, it's far more cost effective because of Nvidia's insane markup. That speaks for itself. Nvidia will not drop their margin because it will tank their stock price. it's half of the reason for all this circular financing - lowering their effective margin without lowering it on paper.

If you think they are going to catch up with Google's software and hardware ecosystem on their first chip, you may be underestimating how hard this is. Google is on TPU v7. meta has already tried with MTIA v1 and v2. those haven't been deployed at scale for inference.

And yes, all their competitors are making custom chips. Google is on TPU v7. absolutely nobody is going to get this right on the first try among their competitors - Google didn't.

There is a pretty big moat for Google: extreme amounts of video data on their existing services and absolutely no dependence on Nvidia and it's 90% margin.

I find this all very cool, but why is this useful for game engine scripting. anyone know?

Anyway, perhaps we can chat in the executorch discord.

And you wouldn't happen to know about a torchscript replacement that is currently in-flight that is not based on export?

Its clear from listening to podcasts/interviews, he does not want to say anything to get on elons bad side. Interviewers appear to also not be eager to broach the subject.

So what are your users doing to get around this? Hoisting all control flow out?

That is true, but that doesn't mean Nvidia is not engaging in engineering to intentionally kneecap competition. Triton and other languages like that are a huge threat and CUtile is a means to combat that threat and prevent a hardware abstraction layer.

Let's see if developers sleepwalk into another trap to keep us locked into nvidia's hardware for the next decade.

Those control flow ops aren't even supported on many backends. I know tensor rt doesn't support them for example, at least today.

Removing control flow isn't as easy as you'd think for some. It essentially means ripping large sections out of python and into separately implemented c++.

I am so confused by metas ecosystem. Perhaps others have the same issues. I have mountains of torchscript code. It worked fine for me - had no issues making the python compatible. Torchscript is now deprecated, and the ostensible replacement is torch.export and either: AOTInductor or executorch. torch.export is so limited - no control flow at runtime at all, less support of python than torchscript. It is far more work to hoist all the control flow out of the model than it ever was to make the model torchscript compatible. Feel like meta has moved on, but I'm still stuck in the past here.

I think my real problem is that xilinx pushes the gui flows heavily. It is extremely annoying to configure the mpsoc fabrics entirely outside of vivado. Same thing for using any of their bundled IP.

what he said, but literally the exact opposite of every prediction for me.

Does it actually creates a unified memory pool? it looks more like an accelerated backend for a collective communications library like nccl, which is very much not unified memory.

Might be helpful if they actually provided a programming model for ANE that isn't onnx. ANE not having a native development model just means software support will not be great.

what is an lpuv2

why exactly do you need a deity to tell you to love your fellow man? Do you need god in your life to want to love your children? I think this is not quite right. I don't expect that the desire to create these tools independent of outcome in the valley is simply about greed , and for companies like anthropic, the ability to use AGI fear as a means to drive investment in themselves from VC class that lives the idea of obliterating human labor. We need less money in tech - we'll probably get it soon enough.

I watched half of those and I haven't had Netflix in 5 years. it's not worth it anymore.

yeah this is essentially everything and all this other discussion of corporate structure is irrelevant

Not terribly familiar with the details, but Amazon kuiper has a contract with blue origin and other providers.

How does fex deal with the fact that the memory model on arm is weak and x86 is total store ordering. It seems like would need to hammer performance by putting memory barriers everywhere to handle all cases. Perhaps fex only works when there are well defined mutexes it can gain visibility into? anyone know?

yeah that is correct. The m series chips can turn on total store ordering memory model solely for Rosetta. There's also some hardware extensions to arm to support x86 condition codes in the hardware because it's way more instruction efficient that way.