This doesn't address some of the cultural expectations in IRC (don't ask to ask, use something like pastebin, stick around because it may take several hours for someone to answer you) but it at least gets over the initial hurdle.
I'm a maintainer of a reasonably big open source project and it works great for us, since you can link issues/PRs directly and get a Github activity log right next to the chat.
It also preserves the chat history, which was one of the big reasons to switch over from IRC.
- Process all user-input so that they can't run commands or do other crazyness in the channel.
- All commands and actions other than chatting are to be handled by the client. This also allows for "IRC2" type commands that aren't part of the IRC spec to be handled "in-band" in the channel, but not seen by any users.
- Channels have a built in admin bot that handles client commands, sets restrictions on channel content, manages mods, channel ownership, names etc. This should be invisible to most users.
- The client should enable end-to-end encryption. Maybe public-key so that user identities can also be verified.
- a user using a normal IRC client and logging into one of these "IRC2" channels would basically just see a bunch of users communicating in encrypted gibberish and not be able to participate
- the client should let people post and view links to images, videos, etc. and automatically embed them in the chat client. Embedded media should stay collapsed until the user clicks it, but then they'll see a resized image or an embedded youtube video, or a player around a soundcloud song or whatever.
- people should be able to nominate awesome quotes to "archive" into a subset of the logged channel.
- channels should be listed on an entry page. Let's get rid of stupid sigils like # and other nonsense. Just show the channel name.
- people can establish private group channels (like a hangout) or 1-1 channels, access is controlled by the default channel bot.
- video or voice chat would be handled outside of the IRC server, but setting up the communications could be handled in-band.
Feel free to knock any of these down as bad ideas.