Hacker News Reader

H.264 and most other modern codecs support “intra refresh” to avoid this problem, at the cost of a marginally higher bitrate overall. Think of this as a “rolling keyframe slice” which marches across the screen every few seconds.

http://www.chaneru.com/Roku/HLS/X264_Settings.htm#intra-refr...

I was not aware of this at the time of writing, but it solves a large problem we've been having. Thank you so much for pointing that out.

Edit: I've just tried using intra refresh, and it works pretty well, but the key frame interval is still required.

I would say intra refresh solves a different problem. You still have to wait for the intra refresh to cover the frame before you can start watching properly. That takes just as long as waiting for a keyframe, and needs slightly more bytes.

The benefit of intra refresh is that you avoid having any particularly large frames. If you're using a sub-second buffer, then intra refresh makes your maximum frame size much smaller without sacrificing quality. It's a godsend for getting latency down to tiny amounts. But if you have 1 or 2 seconds of buffer then it's no big deal if a keyframe is an order of magnitude bigger than other frames, and intra refresh is pointless.

Also it's not really a codec thing, it's a clever encoder trick that you can do on basically anything.

Yes, this is a good point. Intra refresh does reduce variability of the bitrate, but the bitrate is still higher than it would need to be if rtcp was supported.

You could do on most things, but it has to be part of the codec to actually work.

If you aren’t aware of this scheme, you can just wait until the next key frame. With IDR there is a lot more bookkeeping to do so you can figure out when every single needed pixel has been accounted for.

And although it is part of the standard, I’ve encountered players that don’t support it even though they are based on ffmpeg - probably because they do look for key frames to seek to or something.

> And although it is part of the standard, I’ve encountered players that don’t support it even though they are based on ffmpeg - probably because they do look for key frames to seek to or something.

I think that supports my point. It doesn't matter if the technique is explicitly listed in the codec or not. You need clients that will tolerate infinite P-frames, and they will work equally well whether you're using intra refresh techniques on h.264 or h.262

The dumbest possible renderer would have full support; it takes extra logic to get in the way and break it.

Thanks for the easy summary.

One thing to consider: for some IRL performances, it's not uncommon that if you arrive late, you might be seated at the timing discretion of an usher. I understand digital experiences may carry different expectations, but I could see building an experience around this, perhaps starting with audio-only and maybe even a countdown to a next keyframe event (every minute?) while a "please wait to be seated" is shown.

Incredible -- "Our digital usher is finding you a seat in the cloud" slapped on a screen might just save us months of planning and millions on infrastructure.

If you might be hiring for any product roles with the millions saved, feel free to reach out. :)

(The quoted paragraph is no longer in the article. But I'm still curious about it.)

> Therefore, the parameter -force_key_frames expr:gte(t,n_forced*4) is needed, which produces a key frame every 4 seconds.

How often would you like it to be producing key frames? My video experience is mostly with security cameras, and the ones I've used produce an I-frame every 2 seconds by default. Their encoders don't seem to be real high-quality; sometimes there's visible pulsing where the image will get worse until the next I-frame, so I wouldn't want to increase the interval much beyond that.

> WebRTC solves this problem using the RTP Control Protocl (RTCP). When a new user consumes a stream, they send a Full Intra Request (FIR) to the producer. When a producer receives this request, they insert a keyframe into the stream. This keeps the bitrate low while ensuring all the users can view the stream.

I'm writing an NVR that does live streaming with HTML5 Media Source Extensions, sending one-frame .mp4 fragments over a WebSocket connection. My approach to this problem is different: when a new client connects, I send them everything since the last key frame. IIRC there's more data in the I-frame than in the (up to 2 seconds of) P-frames since then, so this seems to work pretty well. If there were only an I-frame (say) every minute, I'd probably look at that inserting a keyframe approach...there is an ONVIF command to insert a key frame IIRC.

As a reference, I think Google Chrome sends a key frame every 90 seconds by default

A few random videos I pulled from youtube had a key frame on average every 4.5 seconds, ranging from 0.1 to 5.5 seconds apart, seems pretty consistent regardless of type of video, at least for the few I tried.

The worst I've seen in production was a poorly configured encoder that insert a keyframe exactly every 30 seconds, along with segments every 10 seconds, which surprisingly, caused some players to crash trying to find a keyframe.

Still sounds better than a FIR. If you consider a big streamer with thousands of users. Users are constantly arriving and leaving, so the keyframe requests are going to be so constant that I can see keyframes being generated much more often than 4 seconds (assuming I understand it all correctly).

Usually you’ll have an SFU between the users and the streamer that can limit the number of requests to one every X seconds.

I guess it comes down to latency requirements?

I would expect where latency isn’t a huge concern, the best user experience would be to start the new receiver back at the last keyframe and fill the buffer up to “present” so they can start watching instantly, and keep a few seconds in the buffer for stability.

In more latency critical streams where you still want the perception of instant video startup I suppose you would have to start at the last keyframe and then as soon as the next key frame came through you could just jump ahead.

"In order for users to watch the video, they must be able to download it in real time, so the maximum bitrate has to be lower than the slowest connection among your users."

Why not just (progressive) download, watch/save, delete? IOW, playback from saved file.

Better for variety of conditions, e.g., connection might be slow.

Streaming really is just downloading without the saving part.

I'm sure this could be implemented if someone were to sit down and implement it.

Ok, so only a problem in live streams?

(And I suppose also when seeking inside a stream)

It seems to me that you can't seek with a webrtc stream, as it is at least.

webrtc is just a stream but you can absolutely tell the sending side to seek to a certain point.

If webrtc is your TV then the sending side is your VHS. You can tell the VHS to rewind or forward, but telling your TV to do the same is impossible. It just shows you what it gets.

I meant it's not part of webrtc, but you indeed can implement a lot of things around it.

It's a problem when playing from a non-start, non-keyframe point (which in practice means any arbitrary point). I'm guessing that's what you meant.