HTML5 Gaming Performance Test
scirra.com
scirra.com
This benchmark gets harder the longer it runs. Which is not in itself terribly unusual. It is pretty smooth for the first bit, where the benchmark runs, but right about where things get hard on my iPad Air is when the test stops.
So this raises a question about how the test endpoint was decided. Because of course if you run longer, the performance would be worse, if you run shorter, the performance will be better, etc.
Now according to the previous article [1], this endpoint is exactly 60 seconds, and an average is taken over that entire period except some time at the beginning for JIT. Now 60 is a suspiciously round number to arrive at by trying to hit X many sprites on the screen or numerically deriving some benchmark objective. Probably, somebody just eyeballed what a Space Blaster game looked like and one minute of runtime was about right.
That probably is a reasonable predictor for Space Blaster, but it is quite the leap to use these results to predict "most other HTML5 games" as asserted in this article. Some games will have half the sprites and others will have double, some games will be more spikey, and others will be more well-behaved, etc.
If I was going to use this benchmark to predict my game's performance, I would do what the benchmark designer probably did. I would ignore the results table, run the benchmark until the sprite count/size looked about right, and take my own average of the FPS indicator around that time.
[1] https://www.scirra.com/blog/107/boosting-mobile-html5-game-p...
It appears to have a low entity count up until the end, and not have any considerable amount of collision detection or other cpu-intensive entity behaviors. This means it's not really stressing a JS runtime and could probably perform great under an interpreter (as demonstrated by Ejecta getting a good framerate)
It also doesn't appear to produce a taxing GPU profile even on a canvas implementation. It draws very few objects, and most of them share the same texture and rendering state, ensuring that even a simple rendering implementation can batch them up into a single draw call with very few vertices. The framebuffer resolution is low and the overdraw is low as well.
A real-world game, even one that isn't 'too complicated' for mobile, would have a dramatically different performance profile.
I think this is a solid benchmark, but it's irresponsible to make claims like 'if sbperftest runs well, most other HTML5 games should run decently as well' and 'The next most common case is usually someone designing or porting such a complex game that even a native engine would struggle on the limited, battery-powered mobile hardware.'
You simply can't make those claims without a broader benchmark suite (which I should note most of the professional mobile/desktop benchmarking suites come closer to offering, with a handful of different benchmarks based on different workloads).
The test is deliberately based on a real-world game (it was adapted from an interactive game) so it's not a synthetic benchmark. No it is not enormously onerous for the CPU or GPU, but if you look at the results many mobile devices still can't quite manage a rock-solid 60 FPS, so I think it serves pretty well to see if mobile devices are fast enough for relatively simple games. There is actually a lot of collision testing in the laser-to-enemy intersection test - all of the objects in the game have custom collision polygons, it's not a simple bounding box test. The explosions use additive blending (or "lighter" composite mode) which actually seems to stress many canvas2d implementations but generally is a lot smoother with WebGL. So it's not that simple a test.
I'm not sure why the 60 second period is important; the most intensive part is the waves of green sprites which should come well within the measured period, and it gets quieter after that when the test ends, so it's not skipping the hard bit.
What I did for my HTML5 benchmarking solution for JSIL games is to instead record the timing in MS for the update and draw stages of every frame. This ensures that even if the benchmark is throttled to 60hz (Chrome does this no matter how you schedule ticks) you can compare timings across browsers on the same configuration and tell whether things have gotten faster.
You can run the benchmark for yourself here: http://www.playescapegoat.com/?replayURI=/stages2and3.replay...
Click 'Show Log' to see the timing data. It's been a very useful tool for evaluating improvements in JITcode quality, GC performance, canvas performance, and various JIT regressions for complex JavaScript. Just running it periodically has revealed dozens of regressions in Firefox and Chrome.
Hopefully with more usable data, your benchmark will contribute to spotting regressions too :D
P.S. feel free to harvest stuff from the JSIL benchmarking code. Everything is open-source.
This tells you nothing about when the frame actually makes it to the screen. It's ok for optimizing your own code, not so useful for comparing across devices, platforms & browsers.
Note that a browser can also run rAF ticks at 60hz even if it's dropping half your frames to hit 30hz vsync. You won't be able to tell.
It is the case that some browser compositors fuck up the presentation of frames (hi Chrome), and that sucks, but you can't programmatically detect it.
I landed a fix here: https://trac.webkit.org/changeset/168549
Also, keep in mind that high throughput - necessary for delivering a consistent 60FPS - will naturally reduce input latency by reducing the length of frames and the amount of time spent not pumping input messages.
P.S. It's my understanding that gamepad input in both Firefox and Chrome is not buffered in any fashion, so that's going to be low-latency.