Strata, Radiance, Maya, etc have really changed that IMO. I use an old X99 setup with 2x A4000s and get a solid ~70 tk/s decode, ~2k prefill, and with 128GB 2133 DDR4, I have memory for quite a few sessions worth of cache so multiple agents can run with decent switching. Probably other performance nobs to setup, but I've found this is great with Qwen 3.8 Flash Next IQ3_S .

Yes, hardware has gotten insanely expensive, but software is moving so quickly and a lot of effort is pointed at eking out every bit of performance, and we are seeing a lot of success. Really looking forward to Qwen 4+. Yes Opus 5.5 is better than what I can use at home, but the way I use these models, I don't feel I need that. I prefer to be in the drivers seat, and these delegate the simple tedious parts to models which they are getting really good at.

I have real world data here. On a 5060 Ti (16GB) with 64GB RAM, I’m getting ~65 tok/s on decoding and ~450 tok/s prefil for Qwen 3.8 Flash Next. 256K context.

This is totally suitable (speed wise) for coding.

This is not some top of the line hardware (might be now, but it should be run of the mill for people that built their system <1.5 years ago).