What you're referring to with "intersex" is actually a set of disorders of sex development which affect each sex differently. For example, consider 5-alpha reductase deficiency: a mutation in the gene that encodes the enzyme for converting testosterone to dihydrotestosterone, causing loss of enzymatic activity, may be present in anyone of either sex, but will only impair male sexual development.
None of this is controversial amongst biologists. It's fundamental to understanding sexual reproduction.