Hacker News Reader

Comment by anon4 | Hacker News Reader

100ms is not that high, that's less than what you get when you're in Europe and want to play with your American friends. And the issue is with delayed retransmission when you can actually afford to send the packets again preemptively.

I would still agree that you should have a bit in the TCP stack that says "use preemptive retransmission instead of waiting so damn long". Can you actually tune the wait time for the ACKs?

View on HN

Yeah, but the 100ms/1% version is more or less acceptable - some client-side smoothing would completely conceal those artifacts. It was at 250ms/5% that things really started to fall apart.

I'd say that 1% packet loss is high, and the most likely reason why you'd have packet loss rates that high is because your network link is congested - in which case, sending more packets is the exact opposite of what you want to do.

High latency, high packet loss, spare available bandwidth. I can think of lots of scenarios where you have two out of three of those things. I can't think of any where you get all three, and this idea only seems useful when you have all of them.

I don't want to play a multiplayer game with hitches every few seconds. You can't fix this with "smoothing" with deterministic lockstep model but you could fix it by increasing the playout delay buffer from 100ms to 250ms, but that's adding another 150ms of latency to the player experience which is less than ideal. It's a much better option to use UDP in this case to get the best experience for most players.

Satellite or radio based ISPs, mobile networking, and if we get exotic, interplanetary communications all come to mind as environments which fit all three criteria.

Interesting - I'm not sure that mobile networking really has massively overprovisioned bandwidth, but satellite and radio links might. Can you think of a way to identify these links so that similar techniques can be deployed when they are found?

No, cellular data is not massively over provisioned; but in the era of LTE, I frequently have more bandwidth than stability.

Also, this scheme is not really asking for massively over provisioned lines; we're looking at fairly small data sets with infrequent bursts to larger sets, and an ack from the receiver knocks the size back down again.

That said, if I were implementing this, I would probably implement a sliding window for messages I send (waterfall encoding would work great here, though it is patent encumbered), and add the ability for the client to request a set of messages it missed. This would make the traffic a bit more predictable and offer the server & client the ability to make up for traffic which was lost during its send window.

You're not sending more packets, but slightly larger packets - as the article says, even 120 frames of data takes up 90 bytes. When, even with extreme repetition, you're sending single digit kilobytes per second over the Internet, it's unlikely that your stream itself is substantially contributing to congestion, unless you have some completely broken/saturated link that nobody wants to play on anyway. (One might want to send enough bytes per frame that such a buffer would start to bump up against packet size limits, but 120 frames is ridiculous. I'm not even sure where he got it from, since 60 * 5 seconds timeout = 300, not 120 - and you don't really want to time out after 5 seconds, since transient total outages happen when accidentally going out of range of a Wi-Fi network or whatever. Layer the opportunistic resending on top of a TCP-like slow resend mechanism for the worst case, and then there's no point sending each packet more than a handful of times.)

I implemented opportunistic resending last year for netplay in a branch of the Dolphin emulator. The target audience was a group of people playing a fighting game (Super Smash Bros.) at a semi-high level, so it was essential to keep latency to an absolute minimum. Even a few frames of lag is enough to substantially impede high level play (might be mitigated by prediction/rewinding if I ever implement that, but since fighting games are based on very quickly responding to others' actions, there's only so good it can get); the 100ms from the article, for example, would be totally unacceptable for anything serious. Even if it were feasible to implement client-side smoothing in an emulator, it would still be much superior to actually get the real data in time.

The question is then whether packet loss is common/significant enough in practice that opportunistic resending's effects are visible in any significant way. For Dolphin, the main advantage of using UDP over TCP is actually NAT traversal, so there is no reason not to do it, but it would be nice to know whether it helped. Unfortunately, I don't have data here: I didn't include code to explicitly record packet loss statistics or anything, and I haven't had the time to actually clean up the code enough to promote it to master, so there aren't many people using it anymore. Anecdotally I implemented opportunistic resending, there was feedback that it was a noticeable improvement; there would certainly be some confirmation bias in there, but the feedback was positive enough to convince me it was likely a real effect. They didn't necessarily have to be losing a lot of packets; as the article mentions, with TCP a single packet lost makes you wait something like 2*RTT, which is certainly noticeable if your input buffer is set to the lowest possible value that produces smooth play under normal conditions.

Sorry I was doing calculations at 2 seconds timeout.

Lets make it even harder for the TCP truthers. Hold on to your neckbeards.

Lets say that somehow sending 90 byte packets 60-times a second for 2 seconds is bad. Lets also say it would be nicer to have a longer timeout like 10 seconds. I agree.

Here's how to do it. Clamp the maximum number of inputs per-packet to 60 (1 seconds worth). If you don't receive any acks for one second something is clearly wrong. If this happens go into "congested mode". In this mode, send packets only 4 times per-second, and keep resending those 60 inputs until an ack comes back. If no ack comes back within 10 seconds, time out. If an ack comes back, go back to non-congested mode and resume sending at 60 packets per-second.

There you go. Congestion avoidance over UDP. Problem solved.

What's a "TCP truther"? Is it hiding under the bed?

> There you go. Congestion avoidance over UDP. Problem solved.

Yes, that sounds entirely reasonable. Please do this, instead of what is described in the article.

The example described in the article sends an average payload of 6 bytes (smaller than UDP header) and converges to a steady state bandwidth that is a function of RTT and the number of player input changes per-second that is generally independent of the amount of packet loss. Given that with this protocol the average average stream is 12kbps, the vast majority of which is UDP packet header, I think your concerns about congestive internet failure are somewhat overblown.

> somewhat overblown.

Somewhat? Possibly one of the most overblown statements I have ever seen. Just try to picture the headline - "Halo 57 multiplayer brings whole ISPs down due to UDP design DDoS". Hilariously overblown.

If we are to be concerned about 12kbps, just wait until my next article when I describe how much UDP traffic is blasted out by non-deterministic game networking models ;)

Reply on news.ycombinator.com