IRC has decades of vulnerabilities and unexpected behavior atop its simple protocol.
Sitting on irc with 100 channels full of randos,
Emacs modules that aren't irc but also process text coming through for X or Y,
misconfiguration,
mime link handling,
irc protocol handling,
sloppy bot making "hey the builtin pretty printing for python code snippet had more implications then I anticipated",
"stuff" running in emacs in between network and the irc client,
add that to "noone seems to talk about or care about security issues in Emacs",
yeah a big pile of risk that pretty tough to just write off, for me, to that point I think it's bananas.
Especially when compared to the trivial alternative of sandboxing another client that has much less surface then a fully configured single process Emacs.
You are doing what you are doing and on some level we aren't too far apart in viewing the safety of a sandboxed raw emacs+some tight set of irc+a convenient plugin X and Y vs another irc client sandboxed. But I bet you aren't doing that.
I really dislike the words "threat model" in security conversations like this.
Neither of us has actually done any of that formal process for this, so why ask for it?