For me it was the ports system; they were a very cool idea at a first glance but when they broke.... and that did happen a lot.
Then RHEL and clones came and that was the end of the FreeBSD for me: I could finally make system updates without heart palpitations and abi/api changes.
A proper package management system and binary packages might have saved FreeBSD in the data centre; maybe it could still do, though it's a very long shot now. It's a shame really...