One reason why the PI can be slow here is lack of any ARMv8 Crypto Extensions and by running in 64 bit mode it can make it even slower.
https://www.raspberrypi.org/forums/viewtopic.php?p=1705614#p...
https://www.raspberrypi.org/forums/viewtopic.php?p=1705614#p...
The 64-bit versus 32-bit thing is almost certainly software wonkiness, as that forum post indicates. There's no reason the 64-bit mode should be slower than 32-bit mode for that task, and benchmarks often show that 64-bit ARM is actually measurably faster than running in 32-bit mode, due to the additional available registers and whatnot.