Please elaborate what you look for:
* Squeak allows access and extension of VM internals: https://www.hpi.uni-potsdam.de/hirschfeld/publications/media...
* Similar for SmalltalkVM: https://hal.archives-ouvertes.fr/hal-01883380/document
* Macro assemblers/High level assemblers are extensible through macros: https://en.wikipedia.org/wiki/Assembly_language#High-level_a... Some assemblers even support algebraic expressions.