So. Maybe we could help other people figure out why VRAM is maxing out. I think it has to do with various new platforms leaking memory.
In my case, I suspect ollama and diffusers are not actually evicting VRAM. nvidia-smi shows it in one case, but I haven’t figured it out yet.
Hey, my point remains. The models are going to get too expensive for me, personally, to run locally. I suspect we’ll default into subscriptions to APIs because the upgrade slope is too steep.
I used copilot to refactor that, and it just didn’t put no_grad back, and I did not notice.
I was uselessly recalculating all of my weights to /dev/null and waste heat.
I daemonized my process to control it more tightly, but I still see way above expected vram allocation.
Just keep in mind. It’s not like the models are going to get smaller, past the quantization limit. What, is there a quantization of retrievable information to 0 bits? ;)