AVX-512 severely downclocks all cores of a processor.

If I'm using some of my cores to run AVX-512 code in the background, it's going to drastically slow down everything else I'm doing.

This can be a significant issue even if you are only running the one task. If you have intermittent AVX-512 code, any time any one of the cores starts running that code, everything else also slows down too which can result in a net loss in performance. Even if you only ran a single core and it used AVX-512, this could still be the case because it takes precious time before and after to lower and raise the clocks that much and everything else runs slower that whole time in addition to running slowly while the AVX stuff executes.