libavif (and the libraries that back it) have a speed parameter to control the CPU/quality tradeoff. Have you tried tweaking that and seeing if you get acceptable speed?