I'm developing framework [1] in Golang with this goal in mind :) It successfully runs relatively big LLM right now, and diffusion models will be the next step

[1] https://github.com/gotzmann/llama.go/