Regarding his wider point, I think something he's not quite taking into account is how personal language is. Yes, we should always keep in mind the LLM is a machine, but because we're human it's also fine to say "please" and to otherwise "play into" the human-like text that is being outputted. Many children still hug their stuffed animals, knowing that they are inanimate. Adults have been speaking politely to Siri and Alexa for years, which do not give off any vibes of possibly being intelligent or sentient. Purposefully adjusting the way you speak just because you know the thing you're speaking to is inanimate is actually what's strange, and human beings are creatures of habit. There is nothing wrong with speaking respectfully to an LLM even if you are under no delusions that its feelings can be hurt. There is no need to speak curtly or rudely just to remind yourself of this.