I haven't tried that but https://github.com/abetlen/llama-cpp-python and https://github.com/r2d4/openlm exists