I would guess the processor would just cause an exception/interrupt and it would just call an OS level exception/interrupt handler which would probably tell the user what the exception/interrupt was that occured; in this case an unsupported instruction.

I believe most processors have undocumented opcodes. Often used for internal debugging or similar.

You might be interested in sandsifter [0], which does precisely that!

[0]: https://github.com/xoreaxeaxeax/sandsifter

Your program will crash with SIGILL.