What part of that comment did you think might sound insensitive - or was that a preemptive worry before you wrote the comment and you ended up not needing it? Either way, didn't feel insensitive at all.
Worth noting that Fry is definitely aware of what Hitler did to gays, but what he did to Jews is higher profile, most known and most understood, so making that comparison is the easiest one in a letter like this.