libaom is slow because it's fundamentally a research product - the folks hacking on it are mostly concerned with the workload of encoding a basket of twenty short clips and demonstrating a tradeoff between Bjontegaard scores and increased encode time, and much less on holistically making a usable encoder.

Source: wrote part of the AV2 spec and traded off a 0.3% improvement in compression for a 1% increase in encode time

True. From my testing SVT-AV1 is about 1.4x faster than x265, and about 540x faster than the original AOM encoder.