Dude - I know you meant it as a joke but it is still uncharitable.

It's hard to appreciate the other party's POV unless you've lived through the stigma of prejudice that's perpetuated by jokes such as yours.

That said, it's okay. Your comment shows you didn't mean any malice so as an implicit apology, it is accepted.