Would you worry as much about coffee? In The UK tea is a commodity. Google "builders tea". It is unusual to drink "black tea" (any Tea made without milk.)
Black tea with milk is the traditional way of drinking Tea in the UK, so no - I really don't think most people give a shit about what you think. It's a cultural thing and would be like me criticising an Indian for eating a hot Curry or a German for their strict beer making codes.