I still don't understand why would you prefer raw memory manipulation to bytestring manipulation. If you want, just make a Swift library that will implement the memory like you want but without unsafe raw access (but just a few methods over a byte array). Back in the days when I did CTFs, I used Python for writing binary exploits, never C.

https://github.com/hellman/libformatstr

You can do something like this, no need to work with raw memory.