"But then I would just resort to using something like quilt to manage patches onto of SVN, which is ridiculous."

This is exactly the way Mercurial patch queues work:

http://mercurial.selenic.com/wiki/MqExtension