freetype definitely doesn't use the GPU and it's the most used font rendering library on Linux. What libraries did you check?