Sitting on irc with 100 channels full of randos,
Emacs modules that aren't irc but also process text coming through for X or Y,
misconfiguration,
mime link handling,
irc protocol handling,
sloppy bot making "hey the builtin pretty printing for python code snippet had more implications then I anticipated",
"stuff" running in emacs in between network and the irc client,
add that to "noone seems to talk about or care about security issues in Emacs",
yeah a big pile of risk that pretty tough to just write off, for me, to that point I think it's bananas.
Especially when compared to the trivial alternative of sandboxing another client that has much less surface then a fully configured single process Emacs.
You are doing what you are doing and on some level we aren't too far apart in viewing the safety of a sandboxed raw emacs+some tight set of irc+a convenient plugin X and Y vs another irc client sandboxed. But I bet you aren't doing that.
I really dislike the words "threat model" in security conversations like this.
Neither of us has actually done any of that formal process for this, so why ask for it?