It's controversial because of several factors, of course. There's the worry that parents, no matter how smart, interesting and committed to their kids, may not be the best at teaching them things. There's the worry that an organized curriculum provides a shared social vocabulary that unschooled kids may lack. There's worries about social interactions with other kids that might be lacking, and so on.
There may be a kind of adverse selection effect, where people inclined to pull their kids out of school may have the most to lose by doing it. So it is worth thinking carefully about the objections and talking about the particular situation with trusted parties to make sure it's the right choice.
I have one child currently in traditional school and try to implement some of the unschooling concepts at home -- in whatever limited time we have. There are definitely positives and benefits to both approaches.
Rather than the conventional approach of defined courses / curricula, it encourages exploration of activities initiated by the children themselves, and learning through natural life experiences.
For example as a result of something the child experiences, the child may have a sudden interest in Greek Mythology, which would then be explored at that point.
While its certainly a form of home education, it is only a subset of homeschoolers that follow this method; many (most?) homeschoolers continue to follow structured curricula similar to conventional schooling.