There are definitely people it matters to, I sometimes get tired of the usual remarks and end up awkwardly using "they" or phrasing tricks to try to avoid them