You can write an extension to support LoRA (~10 lines of Python HF Diffusers code).

If you get to this before me, please create a PR!