I think the reason is very simple: the name. Every time you talk about "real" numbers and "imaginary" numbers, you are telling people implicitly that there is something less legitimate about the imaginary numbers. To put it another way, you're saying that they're not real, when all you really mean is that they're not Real.
Some people just get it quickly: names are just labels, and they don't matter. Others take a long time to realize that, and some never do. Some of us are hard wired to read into the names we give things.