I agree that the “social hazard” aspect of llm objectionable content generation is way overplayed, especially in personal assistant use cases, but I get why it’s an important engineering constraint in some application domains. Eg customer service. When was the last time a customer service agent quoted nazi propaganda to you or provided you with a tawdry account of their ongoing affair?
So largely agreed on the “social welfare” front but disagree on the “product engineering” specifics.
With respect to this attack in particular, it’s more interesting as a sort of injection attack vector on a larger system with an llm component than as a toxic content generation attack… could be a useful vector in contexts where developers don’t realize that inputs generated by an llm are still untrusted and should be treated like any other untrusted user input.
Consider eg using llms in trading scenarios. Get a Bloomberg reporter or other signal generator to insert your magic string and boom.
If they just had one prompt suffix then I would say who cares. But the method is generalizable.