What's happening here is you can make the slack AI hallucinate a message that never existed by telling it to combine your private messages with another message in a public channel in arbitrary ways.

Slack claims it isn't a problem because the user doing the "ai assisted" search has permission to both the private and public data. However that data never existed in the format the AI responds with.

An attacker can make it return the data in such a way that just clicking on the search result makes private data public.

This is basic html injection using AI as the vector. I'm sure slack is aware how serious this is, but they don't have a quick fix so they are pretending it is intended behavior.