I don't think AV1 will ever get to the encoding speed of a codec like h264, because in a very general sense; simpler math is easier to do. AV1 can encode the same information into fewer bits, but that efficiency has a computational cost.
It really depends on what you're doing. If you're trying to livestream it could be a big issue. If you're going to encode a video once and then store it for years, it's probably not a big deal. Totally up to you. For me AV1 has become a pretty good option.