I figured it out, it's because Debian enables PIE and that somehow causes GCC not to be able to satisfy its own rules for allowing inline-assembly to modify %rsp to the value required by this program.
What i ususally do is create a cross compiler with nothing enabled for these kinds of projects. This saves a lot of sifting and disabling options and optimizations.
If you have a bunch of inline assembly and such things, optimizations can be really killing.
The sifter isn't terrible intresting it self but could use a lot of chances. Like the ability to use multiple disassemblers. God that was such a pain to hack that one together.