It doesn't; I leave it to Apple's code to handle the encryption aspect. That said, I'd imagine that it uses PBKDF2 to derive a key from the password. The content itself is encrypted with AES-256. Here's what `man hdiutil` has to say about it:
-encryption [AES-128|AES-256]
As of 10.7, the default algorithm is the AES cipher running in CBC mode
on 512-byte blocks with a 128-bit key.
--Not sure why I'm being downvoted for sharing this. Go figure, HN can be mean sometimes.