OP should have a look at ZFS. With large amounts of data I feel that ZFS snapshots might be far more time efficient to compare than diffing full directories.
Bonus: FreeBSD is currently considering adding Rust to their base system. They have ZFS natively in FreeBSD already. Perhaps OP will find joy in FreeBSD :D