I'm pretty sure there will be no overhead from the type class when the appropriate SPECIALIZE and INLINE pragmas are added, and compiled with -O2. I haven't taken the time to optimize it yet because I haven't had performance problems so far.
Your Vector implementation should offer roughly the same performance. It might be an unnoticeable bit slower because Vector does bounds checking, and my Memory module doesn't (out-of-bounds access is impossible because of the ADT I added).
One thing I'd like to comment on is that I think you should revise your use of unsafeThaw/unsafeFreeze: you don't easily get guarantees of determinism when using these unsafe functions.