I have more experience with SecuROM 8. Did you guys know there is a hidden flag that is parsed on executable running that initializes a separate application which generates a file you can send to the authors of SecuROM that holds the version of SecuROM and quite a lot of other details?
Opaque predicates, VM, simple code integrity checks(int 3), xchg used as control flow obfuscation? It was all there. It took me a while to get the program running under a debugger. Once I did there were a bunch of threads running which would detect tampering or pausing of the threads and kill the process. After suspending those I could do whatever I want.
The DRM used a lot of control flow methods, especially hardware breakpoints and UD2 instruction.
At the time of Unicorn engine's popularity I made a custom plugin to emulate the zlib code embedded inside to search for hidden API calls, unfortunately due to the way QEMU emitted some instructions for the helpers some instructon hooks were called a few times more e.g those with a rep prefix severely skewing instruction counting.
Anyway, VM hunting was difficult. The VM had quite a few single-threaded VM contexts so it could run a lot of programs. From then on it was a matter of reverse engineering the opcodes. at the time I used Olly to trace and log the instruction handlers and opcodes and then using Notepad++ magic I could clean it up from the control flow in a single coherent picture.
Honestly I've forgotten most of it now.
But a few years ago I did write my own Control flow graph generator in Java just to find the 20 or 30 thousand code integrity checks and trace them statically and then was going to use it on the VM.
The most unusual thing is, I often communicated with one of the devs of SecuROM on IRC, of course I did not ask him anything nor did he reveal anything, that is of course illegal. I remember asking him about the little "Cut my life into pieces" tidbit he left just after VMEnter.
Needless to say it is thanks to SecuROM that I gained an interest in RE. Yes I was a novice and dived deep down into a commercial DRM. Thanks to it I learned a lot and is helping me even today, yep as I type this I am working on a SH2 architecture firmware.
But back in the day SecuROM inspired me to write this little ugly thing https://github.com/farmdve/TextVM