The problem is that not all humans are decent. The more of them you hire, the more likely one of them is the asshole who will take advantage of flexibility and ruin it for the rest.
If one person in a group takes unfair advantage of flexibility by e.g. only going to the office half as much as the rest, it sure will have an effect on the "other apples".