You also have penalties from context switches, although you can reduce their impact by performing them in a lazy fashion. AVX512 is even worse, of course.
What I meant is that there are more registers to shuffle around at context switch time, when multiple processes use the extensions.