Maybe a stupid question, I'm not familiar with how video ports work, but does the concept of memory swapping exist for video cards? Or is that physically impossible?
Obviously that would be useless for games, but for LLMs it may be an option?
Obviously that would be useless for games, but for LLMs it may be an option?
Llama.cpp allows you to specify that n layers load into the GPU's VRAM, and the rest load into main memory.