I ask as an owner and regular user of several Pi models, including a 4.
With that said, assuming you choose PCIe, acquire a decent HBA, and build it into a decent enclosure, I'm sure it'd work as well as typical low-end off-the-shelf NAS boxes, and wouldn't cost that much more in time and materials to set up.
I have a Raspberry Pi 4 that mounts a USB hard disk and serves files over SMB and Nextcloud. I have been considering reformatting the drive to ZFS or btrfs and booting the Pi directly from that so that I can start taking snapshots. Is this a bad idea?
I’ve looked at buying dedicated NAS hardware before (mostly Synology products) but I’m always deterred by the cost. A low end Synology NAS with drives runs around $500 or $600, which is a huge jump from my little Pi.
For a low-cost NAS you could use a spare computer. My first NAS was my old desktop computer, using the motherboard SATA ports.
That said, ZFS should work on the Raspberry Pi 4 at least in 64bit mode. You probably will want to use Ubuntu as it has ZFS support.
If you have a spare drive you can test with, give a whirl. Just keep in mind this[1] if you have poor performance from the USB-SATA.
[1]: https://www.raspberrypi.org/forums/viewtopic.php?t=245931