They do encryption/compression on subvolume level; each share you create is a separate subvolume.
For RAID5, they are using it on top of LVM, but with some modification - the synology implementation hooks LVM and btrfs together, so it gets ZFS-like properties.
There's a guy on the internet, who was playing around with it: https://daltondur.st/syno_btrfs_1/