Does this support int4 tensor core operations on the Nvidia Turing & Ampere architectures? From what I've researched this would be a huge untapped speedup and memory saver for inference, but it's mostly undocumented, unsupported by pytorch etc. It'd basically be like llama.cpp but with a 10x or more GPU speedup, if someone was willing to dig in and write the CUDA logic for it. Given how fast llama.cpp already is on the CPU, this would be impressive to see.

I've been tempted to try it myself, but then the thought of faster LLaMA / Alpaca / Vicuna 7B when I already have cheap gpt-turbo-3.5 access (a better model in most ways) was never compelling enough to justify wading into weird semi-documented hardware.