someguy2026··on Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPUDRAM speeds is one thing, but you should also account for the data rate of the PCIe bus (and/or VRAM speed). But yes, holding it "lukewarm" in DRAM rather than on NVMe storage is obviously faster.
someguy2026··on Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPUMy impression is that that is limited to assets and really needs to fit into the DirectX framework. From what I can tell, the gpu-nvme-direct is mostly similar to https://github.com/enfiskutensykkel/ssd-gpu-dma and https://github.com/ZaidQureshi/bam