Designers, people who make their living giving life to a "look" or a "user experience" feel pretty strongly about their work (this is typical of many artists, programmers, architects, musicians, Etc.) In part because they often see their own work as a reflection of them, and in part, and I speculate here, because the creative process is mysterious and it is easy to feel like a fraud when you don't really know how you come up with something, you just do.
So designers look at Elementary OS and they evaluate its look, its theme, its "presence." And what they see seems to remind them strongly of Apple's OS. I completely understand that because that was my first thought as well and I'm not a designer, I thought "Oh, this looks like they put an OS X shell on top of Linux."
I happen to think the other stuff you did, the performance tuning, the application integration, the "fit and finish" if you will is very impressive and it reminds me of how Ubuntu took something mashed up and made it more cohesive.
How ever the design elements always dominate product discussions, even if they shouldn't. Look at the discussion around the Honda Element (car) are they about its innovations in how it converts its internal spaces? Its motor design? No, people talk about how stupid it looks to them (which completely misses the point of the car).
What I'm saying is the Elementary team is going to get razzed a lot about the design elements. Try not to take it personally. The correct response is the one you've latched on to, "We built an OS we wanted to use based on technology we wanted to use, we built it this way." Own the vision.
[1] "rip off" sounds bad "homage" sounds good, both mean "it looks like."