I highly doubt that building a 10 c++ files was as fast as now. I am clearly remember building c# apps for a much longer, i remember how deployment of a trivial app on Azure took ~15 minutes. Meanwhile we still don't have 5k displays with 120hz, hardware is not even here.
Just FIY, you have whisper.cpp in pipeline, but be advised that it is for some reason barely works. tiny model using transformers.js running via wasm outperforms any model of whisper.cpp. fast-whisper is much better here and as fast.
This is basically this - it can learn ignore some paths, and amplify something more important, then you can just cut this paths without sensible loss of quality. The problem is that you are not going to win anything from this - non-matrix multiplication would be slower or the same.
it is barely relevant to big players, but is extremely valuable for small players: distributing your workflow to multiple gpus manually is not that simple thing to do and there are a lot of much more interesting/important problem to solve than shoving your model to a gpu.
This would end up memory-bound for LLMs (and you would better get x3 4090 for this) or by compute for diffusion models (and again 4090 is like 10% slower than H100 for 1% of the price).
it is 20-30% slower than 4090 or H100 in compute, the only improvement is slightly more RAM. This card is not for ML (on purpose) - it is for more enterprisish tasks: some advanced video streaming/rendering, virtualization, etc.
I just tried a lot and the best thing you can get is to do something practical (most ML is empirical anyway) and pick something that you can train on small machine. I picked working with audio since it usually don't need too much data, big networks and can be trained easily on a single 4090.
Does anyone knows good alternative with GPU passthrough for homelab? Hyper-V is not going to fly since it won't do passthrough on consumer windows. Also not proxmox.