For the purpose of one to many type of live streaming you would probably want to use HLS.
Twitch uses it's own transcoding system. Here is a interesting read from their engineering blog [0]
[0] https://blog.twitch.tv/en/2017/10/10/live-video-transmuxing-...