I know some of its features (DTrace and ZFS) are quite impressive but they are not really deal breakers. A kernel is great but it doesn't do much on its own. You still need applications and drivers to have a complete system.
In plain english, OpenSolaris (the right version at least, avoid de-duplication) is the only free platform I'm aware of that will give you a fast, stable, iSCSI server with a file system that's basically impervious to corruption.
The only one. Seriously. COMSTAR and ZFS are that big of a deal. They aren't "nice to have"s. They're solutions that allow you to spend a couple grand on a box, put a lot of RAM in it, attach a JBOD, shove in a few SSDs for read-cache and write-logs and you can duplicate the performance million dollar storage systems on the cheap.
You can do this with very little effort on commodity hardware. You can start as small as you want and scale as large as you want. You get snapshots, clones and replication with barely any effort.
Sun's whole storage strategy (Sun Unified Storage) was built on nothing more than exactly this with a pretty Web interface sitting on top.
OpenSolaris is a real pain for a lot of things, but when it comes to storage nothing else comes close (that I'm aware of).
Don't be scared that you'll lose an alternative to Linux. OpenSolaris isn't any good as an application host if you're expecting anything close to modern GNU tools. Be scared that you'll lose an alternative to NetApp.
It's the only OS that can boot from a ZFS volume and that also has a sane package management system.
In Red Hat-ish distros, I can do "yum update". In Debian-ish distros, I can run "aptitude safe-upgrade". On Debian, I can even do a dist-upgrade and update the box to a new release without having to reinstall it (or boot into the installer). Everything is downloaded, installed and, if needed, restarted. Very close to zero downtime.
For one, you can find the libraries you want and they actually work.
Postgres 8.4 on FreeBSD? No problem. With plperl? No problem. OpenSolaris? Well, if you want to run 32bit Postgres on a system with 48GB of RAM sure. No 64bit for you though. And Oracle shutdown the Postgres build farm servers they were hosting.
Just because FreeBSD stores it's package information in makefiles and pkg stores it in a catalog doesn't change the end result. Ports are as easy to use (if you bother to read the very simple README), and offer far more robust support and packages than any flavor of Solaris I'm familiar with.
There was a time I'd have agreed about Ports, but 30 minutes later I had read the README and realized I was just being lazy. ;-)
http://hackerne.ws/item?id=1567463 (or scroll down)
I hadn't heard of a FreeBSD kernel issue, but if that's true, it might just be for the best for the next 6 months at least. de-dupe is kinda a useless feature if you're not a hosting company anyway. A hosting company could take advantage of storing tons of mostly-the-same VMs. It's not the few hundred GB in our dozens of base VM images that concerns me though. It's the terra-bytes of multimedia and databases.
Plus de-dupe takes a lot of RAM. A lot. There was a time where it would've been very nice, but as it stands, it's really a niche product to come up with a situation where it's RAM requirements, and it's rather small storage savings might make sense.