Of course it's concerning, or at the very least, it's relevant! We get tied up in these debates about motives, experiences, what makes something human or not, etc., when that is less relevant than outcomes. If an LLM, by way of the agentic capabilities we are hastily granting them, causes harm, does it matter if they meant to or not, or what it was thinking or feeling (or not thinking or not feeling) as it caused the harm?
For all we know there are, today, corporations that are controlled by LLMs that have employees or contractors who are doing their bidding.