The facts are that if you have the sex-determining-region Y (SRY) gene (which usually comes on the Y chromosome, but may be missing or defective, or may be on the X chromosome because genes do funny things), you are most likely biologically more-male-than-not and if you don't you probably are more-female-than-not.
But even SRY-or-not isn't enough: XX male syndrome, which normally occurs because of an unusual recombination event where a copy of SRY ends up on an X chromosome, can also happen without a copy of SRY, and XY females can exist with SRY in complete androgen insensitivity syndrome (largely, because of variations in genes responsible for androgen receptors, which are located on the X chromosome.)
The idea that biological sex is binary and corresponds to XY vs. XX is simply false; there's a lot more than “what shape are your sex chromosomes” involved.
Then we have the world of chimerism, ie a person with two set of chromosomes. Genitals of both sexes may be formed.
Then we have mixed gonadal dysgenesis, also known as X0/XY mosaicism.
dragonwriter mentioned XX male syndrome.
Those are just a few big ones. We then have some lovely research into gender identification through sex differences like the size of organs, brain tissue and so on. Works great, except when you do it with transsexuals. For that group the measurements predict the gender they say they are and not their chromosomes. Those studies are used as evidence that transgender is biological, not psychological.
The human body is a complex analog system, and in complex analog systems there aren't much binary 1 or 0. It is more like a scale which leans towards either one direction depending on the sum of a lot of factors. What we call male is just the state when the sum of those factors point in one direction, female in an other. Occasionally you get scales that balance between the two that you can't actually say where it leans.