Have you tried the latest AVIF encoder? It's quality setting seems much more stable.