Fast-Paced Multiplayer (Part I): Client-Server Game Architecture
gabrielgambetta.com
gabrielgambetta.com
It really understates the importance of authority. It blows my mind how many games, especially FPSes, give a lot of authority to the client. Lag compensation for shooting is one thing, but clients should not be able to shoot through walls or speed hack. These types of cheats are easy to prevent server-side, but developers don't because .... ? No idea. I've always guessed that it's to reduce server-side CPU load, but that's all I can guess.
"easy" so you think veterant c++ dev that invented those concepts are dumb or lazy and it's a solved problem?
It's not easy and not solved because those are very complicated issues.
Why is it hard to add server-side detection for line-of-sight when a player shoots at another player? The client should not be telling the server "I shot Player 2 and did X damage". Instead, the client should be saying "I shot in X direction at Y time" and the server should be reconstructing the world at Y time (As described in the Lag Compensation page of the article) to determine a hit. The server should be performing line-of-sight checks.
In a game like PUBG, there's no reason a player should be popping headshots against players a mile away when there's a hill and/or several buildings between them. But the game allows clients to decide when they score a hit. At least, it did in the past.
In anycase it does not work like it should in Valorant.
Client-side hacks that control player input or give information to the player that they shouldn't have (ie, being able to see through walls) is hard to deal with. I'll give you that.
But what is EASY to deal with is the client doing things they shouldn't be able to, like shoot through bullet-proof walls.
And I'm not talking about cases of running behind cover but dying anyways because lag compensation meant that you saw yourself behind the wall, but the other player didn't because what they see is 100 ms behind what you see. I'm talking about cases where you were never exposed from cover to begin with, and the client is allowed to decide if a hit is done or not. In another comment, I linked several examples of players in PUBG scoring hits that were straight-up not possible due to lack of line-of-sight.
Here's a good example: https://youtu.be/2Gn2Sixr5Wk
The type of hack being used in this video is 100% preventable via server-side checks.
Sure, with a huge hill that cannot move, its impossible. But what if instead of a hill, the player is shooting through a truck that moves?
Well, as the server you know you sent the vehicles location to the client at v1, v2, v3... And you know the player was at p1 when he shot. But you dont know at what time the truck crossed between the player and his victim because that depends on when the client received it. You dont even know IF he received it because you sent it by UDP. So maybe the client dropped the v2 packet.
And then remember that the client can rewind history. Basically he can report he did something 100ms ago and the server has no way to distinguish that from a cheat or from lag.
From a practical sense, i think devs are faced with a tricky situation: hacks are always possible. Even if you have perfect simulation, it doesn't prevent aim bots because those are theoretically possible. On top of that, you have to trust the client to some degree because of lag compensation. And running physics simulations server side might not be practical.
So I think it's easy as a dev to sort of give up and try to solve it some other way. Maybe you just hope that anyone doing something this egregious will be banned manually.
The second reason is that you dont want to instantly ban players because it lets the cheaters know exactly the mechanics that caught them. This is why there is always a delay before the real ban happens. Of ourse the side effect is that obliviously cheating players have to be tolerated for a bit before the ban hammer comes down.
The server can do a best effort attempt at rewinding the game state and take into account how the client is interpolating the world, but it gets very complicated as game complexity goes up.
Someone correct me if I'm wrong, but I believe the Source engine does that for the attacker and the victim, but ignores all other entities for the ray casting (so it can only really prevent shooting through the map walls)
Hiding lag requires guessing about what moving objects are likely to do. This works best for systems with a limited set of moving objects, such as players and bullets. As the author points out, bullets are handled as a very special case.
I've done some work on this inside a Second Life / Open Simulator viewer. That's a very general system - anything can potentially move. You can make guns and shoot things, but the system has no idea of "bullet", just physics. The server is authoritative about everything except which way an avatar is facing. So turning is fast, but movement only happens after one round trip to the server. Round trip delay is typically 40-200ms. The problem is hiding that.
The server sends the viewer updates with position/rotation, velocity/rotational velocity, and linear acceleration. That last makes projectiles describe an arc under gravity without server side updates.
When an update comes in, the moving object has to be yanked back to where the server says it is. There are situations in which the predictor isn't very good, especially in the presence of jitter. And, the worst case, a region crossing, where there are no updates for 500ms-2000ms during the server handoff. The original predictor would sometimes put vehicles far into the air or into the ground for one update cycle. The predictor was using the last velocity, which is noisy as a vehicle bumps along a road following the outputs from the physics engine. As in real life, there's vibration. The last velocity, intended to be used only for 1/45 second, was being extrapolated to 500ms-2000ms, or sometimes as much as 100x the time it was intended for. So, low-pass filtering and a error metric was added to make vehicle movement look sane. The illusion can be maintained reasonably well up to about 200ms of lag, but beyond that, it shows as "rubber banding".
Shooters just don't work well in a system this general. Archery is sort of OK.
As the article mentions - fully server authoritativeness works great in lan situations and for single player content but can really fall on its face in multiplayer.
If I, as a user, am constantly moving in random directions and have a lower ping than you (and thus my movements are reported to the server before you can actually observe them on your machine) then you should never hit me except through pure blind luck - in fact accurate shots could be the only shots that are guaranteed not to hit.
Games want to feel bad at a range of reasonable pings - and reasonable pings usually stretch into the range of 100 or 150 ms - that leads to up to a half a second of lag between when I dodge left and you, as a shooter, would have the earliest chance to actually register a shot with the server at my new position.
Additionally, the server can reconstruct what it thinks we saw at a given moment in time - but the server doesn't know what data has been lost[1] and whether the client might think it was right - or even whether the client might be misreporting dropped packets to engineer an advantageous game state. Also, this whole time we've been discussing latency as if it's a fixed number - but it is constantly changing and the reporting of the latency is something you need to trust to a client since clients can always artificially inflate latency figures.
Due to lag correction either you're able to shoot me after I duck behind cover, or you're never able to shoot me - and both options suck for one half of the players involved.
1. See the two generals problem - proof of delivery is actually a hard problem.
Your "behind cover" example describes best your concern with lag comp. It works both ways. You having lower latency can suffer when attempting to take cover. However, you have the advantage when exiting cover; you will see others first.
It's as fair as it can be. As for feel, it definitely feels fairer than no lag comp.
All these factors are pretty heavily interwoven in how the game ends up feeling to players.
It might be, especially with regular (i.e. not pros, and not wannabe-pros) players. Two main sources of latency: shitty ISPs, and using Wi-Fi instead of wired connections. A shitty router (i.e. like the ones given out by most ISPs), a bad placement of the computer, or interference from neighbors - all can induce periods of 150ms+ latency.
> That’s assuming people are connecting to servers in the same region of course but if they’re not that’s a problem of the matchmaking service.
That's assuming the game is already so successful globally that it can afford having servers close to everyone, and that it has enough players at all time that matchmaking services work well.
https://gfycat.com/CalmVillainousDore
https://youtu.be/BnZ_FQKrR90?t=100
I mean, maybe it's changed and PUBG now has server-side hit verification. But for a long time, it didn't.
I’d say that this article handles the speed hack issue because of authoritative server state, but doesn’t go far into limiting what is sent to the player or not. The issue arises when you didn’t make your game multiplayer from the start and adding prediction and sync visibility isn’t an option, that’s where you get player authoritative states that share their states with the other players. Client authoritative is easy to implement, impossible to prevent cheating.
Its so easy to do, but pretty much no game in the industry does it? I guess they must all be incompetent.
In their cases it works because they have simple maps and a low amount of entities like players, so its not too much work for the server to do culling so they can hide positions of other players until its needed. But when in close proximity they can be seen through walls, like around a corner you can shoot through (at least csgo has penetration) and to avoid pop-in. And the other players still produce sound so information about other player positions are still sent to all clients and could be used for ESP.
So no, this is not a solved problem, and is only partly so in very few games.
Game devs on big commercial games are often not seasoned veterans, in my experience. The industry largely runs on the fanatical, underpaid efforts of kids who will do anything to work on games.
For example, in Fortnite, if you toggle a lagswitch (as in, block Fortnite UDP for some time), then reconnecting to the server will return you to the same position you were at when you disconnected, assuming you were just trying to run on the ground. However, if you toggle a lagswitch in a vehicle, the servers don't want to do expensive, complicated physics server-sided, so your client becomes authoritative to your position and the sanity checks are simply making sure you're not teleporting around large chunks of the map in seconds. If you were to hack your client and give your vehicle moon gravity, the server generally wouldn't care about that.
Say you’re driving a car in a straight line, there’s very good odds that in .25 seconds you’ll still be going straight so your predicted line will be very close to the real line. With players you can’t guarantee that, they could zig zag, turn around, jump, etc much more likely to change and require prediction reconciliation.
Straightforward extrapolation as with dead reckoning does poorly with both cases.
Then once you’ve mastered the art of getting cars not looking wonky you have to deal with collisions between them when they’re moving at speeds where typical gameplay latencies make it easy for discrepancies between clients to cause mispredictions which result in complete misses or large interpenetrations that physics engines have a bad time with.
That CoD video explains what you do instead with the client buffering segment. You show other clients in the past even from the point of view of the latest data you received so you can interpolate so get no discontinuities at the expense of a slightly wrong view of the universe. Your client is one round trip ahead and the other clients are one round trip plus whatever delay behind. Which is where things like peekers advantage come from.
Similarly they state exactly what I’ve said that extrapolation ends up making the game unplayable and again a input buffer on the server is the solution. The concern there is how much you buffer as it impacts latency quite heavily. So their system is dynamic based on the quality of the client connection.
Note also in this case that extrapolation is only done in the server which is not responsible for the visual result and is done based on client input data which is usually sent at a much higher rate than the server sends state about other players back. Consequently the visual discontinuities will be hidden by the interpolation on the clients and will be small as mispredictions are expected to be caught quickly and the players are relatively speaking slow moving. It’s interesting they still need to stop extrapolating and fall back on buffering in the worst case.
The reason determinism is needed is that the client is only sending their input so in order to work out where they are from only that information it must play back deterministically on both client and server. The same series of inputs must result in the same movement otherwise they’d end up in different positions. It’s also very useful because you can then rollback and replay movement if a missed input frame arrives.
Rather than demonstrating your point it refutes it pretty handily and is a pretty good lay explanation of the considerations when writing networked gameplay systems. Believe it or not this has been my bread and butter for the past sixteen years!
Pretty much every real game map is going to have hills or elevation changes. You're really never driving in a straight line.
Stadia (and similar) can render entire games server-side and then deliver them to you for a nominal fee. I'm certain that in 2021, we can hook up a nvidia gpu to the server and allow some basics physics calculations run for ~128 players.
If we can't, there 100% is a market for a chip, card, server, etc. that can do this. There is an entire group of gamers (30's+ with kids and demanding job, not a lot of time) who are sick and tired of FPS cheats and just want to relax and play a game for an hour a night. They would happily pay for a non-cheating version of a game.
I've been doing large scale distributed systems for a decade, and I still feel like I'm learning especially as I retool for games.
The way that I'm trying to make this simple is by inventing a new platform for "serverless game hosting" using my programming language (Adama : http://www.adama-lang.org/ ).
My focus on board game is the ultimate in server's role on authority since board games are more transactional between players and staggered (I play this card, that card enables another play to react, that reaction enables another play to augment, etc).
I'm looking into what it could mean to escape the board-game pit, and one thing that I'm thinking about is latency. The question that I'm looking at is whether or not I can have a programming model be differentiated such that the client prediction code can be generated.
A trivial way to exploit my language is to bring edge computing into the mix such that the edge compute runs a copy of the entire game, and then ships deltas quickly which will be overwritten from the authoritative server. This will work on updates to existing data (i.e. moving position), but it will fail at creating new items unless I move unique id generate to clients (which, is less than happy for me).
This makes it an open-ended problem - many of the tricks work under very specific assumptions (e.g. low latency, trustworthy clients), and fail when those assumptions no longer hold (e.g. people notice stutter, cheaters cheat). The job is to find the right set of tricks for your use case. And it starts with knowing the tricks in the first place (or inventing new ones), as it's pretty much a specialization of its own.
What really surprises me is how how often I see this mistake in web development. "No, Bob, you should not increase the user's bank account based on that number passed in from the frontend React app."
It was a convoluted example for examples sake, but I'm pretty much referring to them missing this important check here.
- Generate a transaction number and associate that number with the respective transaction details
- Send this number to the customer's mobile phone with all those details, or other configured device
- the transfer is only authorised if the customer has entered the transaction number
That way, the customer is very likely to have verified the details of the transfer.
This is the standard way online banking has been operating in, I believe, much of Europe since practically forever (although it used to be physical lists before smartphones were popular).
IMO, This is clear-cut evidence that PUBG uses (or perhaps, in the past used) client-side hit checks. Notice there's not even bullet travel time. Just "pop pop" and a target that isn't in sight and never was inside instantly dies.
a few do client side prediction for projectiles too (Overwatch notably)
But they didn't really compensate for exploiting this to do things that were actually impossible. (ie speed hacks)
It wasn't until into the 2nd expansion that I got broadband, and hoo boy was it hard to adjust to. Suddenly I didn't have to anticipate and plan ahead nearly as much, and I had a whole new learning curve to go through to get my timing on point again. It seemed to me at the time that the game was structured pretty decently to handle the huge differences in player system specs and connection capabilities.
Have a watch of this.
If you play a first-person shooter with 300-400 ms ping and the game uses server side hit detection, you're not going to have much fun.
By moving to client-side detection, players hit what they "should hit" from their perspective.
At least that's how I recall it from when COD and BF moved to client-side hit detection.
1. Authoritative server 2. Client accuracy
By 'client accuracy' I mean that was happens on the client actually occured. Ie. If you shoot a player, he actually got shot.
The conundrum is that these goals conflict. They are like Heisenberg uncertainty: you can have position or momentum, but not both. You can have server-side authority, or you can have client side accuracy, or somewhere in the middle.
Just as an example, most shooters have the goal that 'if you shot him on your screen, it was a hit', even if according to server side knowledge the recipient of the shot had already moved. Otherwise when you're shooting you would have to guess where the player is going to be when you're bullet packets reach the server.
So instead, we let the client decide what a hit was. Hopefully the server has some sanity checking.
I shoot at time t, server receives the info at t+s, reporting that I shot at some trajectory at time t.
But the client can erroneously edit history. What if the client sends t-0.1s instead of t? There is no check the server can do to prevent this... I appear like a client which has 150ms of latency even though I really only have 50.
Edit: I see you wrote a cg book! https://news.ycombinator.com/item?id=26013002
https://mrelusive.com/publications/papers/The-DOOM-III-Netwo...
Isn't there a really obvious exploit here? The client could lie about the timestamps and use that to take advantage of future information in making its decisions. For example, in a simplified fighting game where you have two attacks (top and bottom) and two blocks, it could always choose the perfect block to apply before the opponent attacks—simply by seeing the incoming attack packet, and then immediately sending a block packet with an "older" timestamp. This doesn't entirely break the concept of an authoritative server, but it seems to diminish it significantly. I know lots of fighting games use this "rollback-type" netcode, maybe they've developed workarounds for this?
There are two ways you can cheat. Because you have the full simulation state and input, there is no possible precaution against map hacks.
Some FPS games just won't send exact player positions until the client is about to be able to legitimately observe them. Not possible here.
The other way is, as you say, you can build fake inputs.
To do the timing attack as you describe, all of your input would have to be pretending to be slow. You can't send a t=10 I'm idle packet, then a t=8 I blocked.
You'd need a fantastic connection to the server, because the delay you're exploiting is there to protect you from your lag.
Your oponent would also need a fantastic connection, because you aren't seeing their current inputs, you're seeing their past inputs.
My feeling is that it would be very challenging to blend human like behavior with superhuman reflexes when doing automated responses.
But there's nothing you're missing. The attack is definitely possible.
Also I think no one has attempted to cheat like this because it's much easier to cheat with a program that 'reacts' to graphical cues (such auto-block bots are pretty common in the most recent Tekken and Street Fighter entries). There are some very rare moves that are impossible for a program to react to like this (usually '0-frame grabs') which hit on the exact frame the input is entered, I'm not sure what if anything the current cheaters do about those.
He wired up a LED to the mouse button, so it lit up when the button was pressed. By filming both the monitor and the LED with a high-speed camera he could measure the true input lag.
BF4 had quite horrible netcode on launch, but after a valiant effort by Viceral it got quite good, at least on PC. Sadly the game had lost momentum at that point, almost two years after launch, so not many players left to enjoy it.
That was well before the modern shooters kids play these days. I'd say things have not improved dramatically over the last 10 years or so. Sure CS servers now run at 100 or more FPS but other than that, not a whole lot of change. In fact, popular examples such as PUBG suck immensely as far as networking is concerned