Sex is very much a 'binary' concept; it's fundamentally tied to a property of reproduction strategies; the size of the gametes. To quote [0]:
> The type of gametes produced by an organism defines its sex. Commonly in plants and animals, male organisms produce smaller gametes (spermatozoa, sperm) while female organisms produce larger gametes (ova, often called egg cells).
Convention is; if you produce small gametes, you're male, if you produce big gametes, you're female. Sexual reproduction happens pretty much exclusively in two sexes. Biologically speaking, individuals that don't produce gametes, don't have a sex.
To quote from [1]:
> There are almost invariably only two gamete types, all analyses showing that intermediate gamete sizes are eliminated due to selection.
Gender can be considered a cultural construct based upon the interpretation of sexually defined roles and strategies. But sex itself is binary. So "Intersex people" - your words, not mine - do exist but as an defective expression of the reproductive function, see above. I'm not aware of any medical condition in humans that results in functional hermaphrodity as seen in some plant species, that is; individuals that produce functional small and large gametes. Although there is a condition called 'True Hermaphrodity' whereas an individual develops both ovarian and testicular tissue, this does not result in functional gametes.
> Sex is mostly based on a mix of genome
There are many sex-determination mechanisms, not all of them involve different genomes. Some are based on temperature, for example. Again; see [0].
> Honestly I've some trouble understanding how some people find this to be such a hard concept,
I do wonder that myself and I guess it's because for some people this is a placeholder for some kind of ideological discussion, separate from the actual discussion about sex. The impreciseness of colloquial language probably also plays a part. But to summarize;
Gender: cultural construct based on interpretation of reproductive roles and strategies, pretty much unbounded in its expression and shape and form
Sex: "binary" reproductive trait that comes in two expressions; male (small gametes) and female (large gametes), although there are some species where both male and female is expressed in single individuals. Sex doesn't necessarily depend on the chromosomes but in many species can also be decided by external factors.
Hope that helps!
[0] https://en.wikipedia.org/wiki/Sex
[1] https://en.wikipedia.org/wiki/Gamete