ZFS runs fine with just 1GB of ram. Even several terrabytes of storage. When it comes to knowledge about ZFS, avoid about anything that comes from the FreeNAS community. It is quackery at its best.
The problem is that there's so much quackery around ZFS when you Google for answers to any major problem. It's frequently hard to get good information.
I just want to point out that the official documentation is absolutely great.
Doesn't ZFS dedup eat lots of ram? btrfs offers on-demand deduplication instead of it being always-on.
Yes, it does. But dedup is rarely that useful in my experience, it sounds nice on paper but it often doesn't provide benefits worth the CPU and IOPS demand it generates. Compression is "good enough" 98% of the time, and ZFS does a fine job with it.
If you take snapshotted backups from ext4 etc filsystems (no send/recv), where users regularly reorganise photo folders, dedup is invaluable :-)
dedup is quite useful when you have snapshots that somehow have lost their space-sharing. I guess it's due to files being replaced with identical files.
The Btrfs deduplication... also needs vast quantities of RAM. That's just the nature of the beast. Don't use it, and both systems have relatively modest requirements.
but it's on-demand and can use a scratch file, which makes it far more managable, e.g. you can let it run over the weekend.
If you need filesystem dedup... Perhaps handle it in userspace instead
Since ZoL does not have reflink copies or the range-dedup ioctl you cannot do it from userspace like you can with btrfs.
I run ZFS on a VM with 512MB RAM. It does web/mail/file serving and it has worked great.