ZFS has been in production in Solaris 10 since June 2006 and is quite stable and production tested.
btrfs is still considered experimental, even though it's been in the mainline Linux kernel since 2009.
You might look into the KVM support in SmartOS if you want to run other OS's on top of Solaris/ZFS.