Both the library and the AVX512 generating JIT binary would have the arch set to AVX512 in the executable format header.
If the arch is incorrect, it is a bug, and program will crash on illegal instruction. Ie like if you had an ARM binary that was incorrectly set to x64 and was run on x64.
What if the JIT generates new instructions dynamically and initially there are no AVX512 instructions but later on in the process' lifetime there are?