The effect of web browser “Input Lag” in HTML5 games (2017) | Hacker News Reader

The effect of web browser “Input Lag” in HTML5 games (2017)

vsynctester.com

56 pointsHugoDaniel23 comments

Fun fact: DWM enforces >1 frame of input lag on every Windows application running in a window. It is impossible to do better than 1 frame. But DWM itself does better: when you drag windows around they apparently stick to the mouse cursor exactly, which would be zero frames of latency. How do they do it? By cheating: they delay the mouse cursor by one frame, only while you are dragging a window. So the latency is still there but less obvious because it doesn't cause a difference between the cursor and window motion.

> It is impossible to do better than 1 frame.

Exclusive fullscreen applications shouldn't need any compositing, surely they must be exempt from that.

There's still plenty of latency sources on the way from he mouse hardware to the application's event handling code. The remaining way to the display just adds (more or less) latency on top - which isn't as important when it's just about the discrepancy of the mouse cursor and what's rendered beneath though. But the mouse cursor can use all sorts of special shortcuts, for instance the cursor is a hardware sprite which is updated and rendered indepedently from the rest of the "rendering pipeline" otherwise the cursor movement would feel laggy all the time.

> for instance the cursor is a hardware sprite which is updated and rendered indepedently from the rest of the "rendering pipeline" otherwise the cursor movement would feel laggy all the time.

While investigating cursor lag issues on my computer (which were caused by an errant Intel Optane card, of all things...) I read through Windows’ video driver documentation and I learned that while Windows does direct the mouse pointer completely independently of the DWM, there is no requirement that the cursor be rendered during scanout (like a hardware sprite) and it is permissible for a GPU to render the pointer to the framebuffer. In fact, there’s a lot that’s undefined by the specs - for example, I used to believe there was a hard limit of 127x127 on cursor sizes (based on the ICO file format) but it turns out there is no upper-limit on cursor sizes at all. If you set a huge cursor image (say, 1500x1500px) some GPUs do render it to the framebuffer but others do not (in my testing, newer Radeon cards behaved differently if the image was above ~600px, whereas with my Nvidia cards anything over 255x255 had a different mouse-acceleration curve, that was weird)

Yes, exclusive fullscreen mode applications are not running in a window, so they can have zero frames of latency (though almost all don't, for various reasons). I am not sure about borderless fullscreen mode, I think this may also have an optimization that can eliminate the 1 frame delay. But normal windows always get the delay.

There is also a feature designed to let applications bypass this delay in windowed mode using GPU hardware overlays, however it needs GPU driver support and it is not supported on Nvidia or AMD drivers AFAIK. Basically nobody uses it.

MS had a big push to completely deprecate exclusive fullscreen a couple years ago by improving borderless fullscreen to match it.

I'm not sold that borderless fullscreen matches it. I can see the difference in video panning between exclusive fullscreen and borderless. Granted, it is not a huge difference, but it does exist. At the end of the day we continue become further and further abstracted from the hardware. Input lag on the display, input lag in the device, etc.

Video panning differences don't have anything to do with input lag. The only way fullscreen exclusive mode would change the appearance of video panning (bugs aside) is if it changed the mode of the monitor so the refresh rate matched the video frame rate (or equivalently used G-sync or Freesync to do the same).

Madvr on Windows for video playback supports both windowed hardware overlay and fullscreen exclusive mode as options so that is one application which exists.

This may refer to the old style video-specific YUV overlays, which I don't think DWM uses or supports these days. Video playback doesn't need low latency anyway, just predictable latency, which DWM can provide in normal operation.

Fullscreen exclusive mode support is still common in games.

>Fun fact: DWM enforces >1 frame of input lag on every Windows application running in a window. It is impossible to do better than 1 frame

AFAIK you can avoid it by using DXGI flip model.

random page:

>(DXGI) Flip Model is a new mode introduced in Windows 8 that makes the SwapChain work efficiently with the Windows DWM (Desktop Window Manager). Flip Model eliminates performance penalties normally associated with windowed mode rendering and introduces new methods to measure and regulate render latency.

https://wiki.special-k.info/SwapChain

Unfortunately not. Flip model improves efficiency by removing one copy. However, the window contents are still composited into DWM's backbuffer just after vsync, so any rendering that happens during the frame is not displayed until the following frame.

There is a flag DXGI_SWAP_CHAIN_FLAG_ALLOW_TEARING which can completely eliminate DWM's latency even for normal composited windows, but it only functions on systems with hardware overlays, which are not supported on Nvidia or AMD.

> which are not supported on Nvidia or AMD

What about Intel and Matrox?

What about desktop compositors for Linux? And the macOS compositor? (Not that macOS is pleasant to use with a mouse...)

Intel does support it. No idea about Matrox, but it seems like they don't make their own GPU chips anymore so probably not.

I believe the situation is worse on most Linux compositors. MacOS I'm not sure of. I tested just now and it looks like at least one frame of latency on most things, and they don't implement the window dragging hack DWM does either.

I note that touchpads (especially on Macs) seem to have considerably higher perceptual lag compared to mice, I think because of the smoothing necessary to make touchpads less fiddly - and as most macOS users are using a touchpad I imagine that contributes to less complaints about lag specifically because it would be difficult to pin-down.

Honestly macOS has always felt laggy to me in comparison to Windows. Even with a mouse. I think there are latency issues there, but also many design issues like the interminable animation for maximizing windows. Compare to the speed of Aero Snap gestures.

>Not that macOS is pleasant to use with a mouse...

Huh? That should be the first time I hear it itsn't...

Is there a reason DWM doesn't triple buffer? I'm not skilled in the dark arts of graphics, but it would seem to me if you triggered compositing to a backbuffer on vsync and to a separate backbuffer when a window flipped its swap chain, on the next vsync you could use the flip triggered buffer if it finished or the vsync triggered one if the flip triggered didn't finish on time.

If the flip triggered composite finishes before vsync, you could also let a second flip trigger a new composite, etc. If your windows do a lot of flipping between vsyncs, you could burn a lot of cpu/gpu time compositing, but latency improvement would be worth it, at least for some applications.

Edit: I have seen a lot of negative opinions on tripple buffering, but they seem to be related to increased latency as a result of implementations that wouldn't drop rendered frames, even when two frames were rendered between vsyncs. That's a silly thing to do, so I assume a triple buffered DWM wouldn't do that.

They just don't prioritize low latency enough to implement such a feature. They're probably more focused on power efficiency. The overlay solution is better (although it's not free wrt power consumption either). It's just lame that Nvidia and AMD don't support it.

To be fair, it is difficult for applications to support <1 frame of latency even when the compositor allows it. It requires precise timing and very predictable rendering performance. Even VR applications struggle with this, despite the fact that the consequence of latency and/or missed frames isn't just a floaty feel to the mouse pointer, but literal nausea.

Clever.

If we can just teach audiences to clap one sixteenth-note later than usual then all live music could go completely digital!

Mouse-to-display lag is even a hard problem in native desktop applications. TBH in this demo it doesn't look much worse than in a macOS, Windows or Linux UI application that renders through a 3D-API (and doesn't use all sorts of platform-specific hacks and tricks trying to minimize mouse latency).

Can anyone opine as to how these tests are achievable in a browser?

The “game” part at the top measures human lag time as a proxy for compositor lag, which is a crude approximation as it also includes monitor input lag as well.

Other tests, including on the same site, are able to get the real-time hardware cursor position which the test code knows is ahead of the last-drawn cursor position and whatever position it’s drawing right now.

Reply on news.ycombinator.com