A Real-World WebAssembly Benchmark | Hacker News Reader

A Real-World WebAssembly Benchmark

pspdfkit.com

243 pointsMartinMond89 comments

Personal WASM anecdote: I just implemented a WASM based 3d model boolean operation for three.js where I can intersect two models together. The pure js version ran in around 10s on a relatively simple model whereas the WASM version did the same thing in a little under 1s. I've been very impressed by the technology so far! Obviously this kind of CPU intensive work is where it can shine and it's a useful tool to have in your toolkit.

Can you post the code somewhere? I'm very interested in this.

For geometry heavy stuff, web workers (possibly in addition to wasm) be super helpful in preventing the browser from blocking while number crunching.

wouldn’t shader libraries be better?

Ideally yes, but the lack of geometry/tessellation/compute shaders in WebGL limits what you can do on the GPU.

Geometry processing that's more complex than moving vertices around has to be done on the CPU for now.

you should check out http://gpu.rocks/ pretty awesome GPGPU js library.

>Geometry processing that's more complex than moving vertices

Have you heard about our lord and savior abstract vector spaces?

everything is just moving vertices around in linear transformations.

and whats the performance using native code(c/c++)? Im guessing ~0.1sec on same models.

From benchmarks I've seen elsewhere, native code is generally roughly 2x faster than WASM, so I doubt that's the case. I'm sure it depends on a lot of factors, though.

Single-threaded, non-SIMD x86-64 code being 1.2x to 2x faster than WASM is what I'm seeing as well. Interestingly I don't see such big differences between browsers as the PSPDFKit benchmark, my suspicion is that this mainly measures overhead when crossing the WASM-JS border.

PSPDFKit CTO here: We've been using WebAssembly since 2017 to render PDF documents directly in the browser and strongly believe in the future of the technology.

We published this benchmark to have a public indicator of the performance gains WebAssembly promises in a real-world product.

Any chance you also benchmarked WA version against native code? What are results in your case?

We did that in https://pspdfkit.com/blog/2017/webassembly-a-new-hope/ - WebAssembly is really fast, the different is less than you'd expect.

It's still getting faster too, e.g. engines are starting to use trap-based bounds checking which greatly reduces overhead when running on 64-bit platforms.

Could you elaborate on “trap-based bounds checking”, or provide a reference to learn more? My searching mostly pointed to “trap” as being a WASM concept, rather than some processor feature that improves bounds checks.

It means using the processor's memory protection features to prevent programs from reading memory that they're not supposed to read, rather than inserting check instructions at every memory access.

Yeah, the V8 devs go into detail here: https://docs.google.com/document/d/17y4kxuHFrVxAiuCP_FFtFA2H...

They just recently shipped it on Linux with V8 v6.8, but the Windows implementation is still a work in progress.

Excellent post! Webassembly appears to be approximately half as fast as native. Do you know where that gap comes from, and how much of it can be expected to close?

Hello Martin - slightly off topic. There's no pricing section on your website. Does one have to get in touch with sales for a quote?

Yes, please contact our sales team to discuss your use case. We sell to indie developers, early startups up to large Fortune 100 companies with a wide range of use cases so we found custom pricing to be the best way for us to be able to sell our software to everyone and at the same time support our team size.

Hey, thanks for the interesting article, and the benchmark of course!

One bit of constructive feedback: add a date of publishing to the article. PSPDFKit already has multiple articles on WASM before, and it's nice to be able to tell which one is the most "up to date" at a glance.

Hi there,

Thanks for publishing this benchmark, but I've got a question about the results. The description of the charts states that 'A lower score is better', and between the two charts you say 'when we compare WebAssembly with our JavaScript fallback ... all browsers except Firefox are still slower'. But WebAssembly has a lower score than JavaScript on all browser/OS combinations except Chrome67/macOS, Safari/macOS and Edge/Win10. Could you clarify which way the results are pointing?

Firefox crushing the competition on this (all on iMac late 2015):

  Firefox Nightly: 2216  
  Chrome Canary: 5078  
  Safari Tech Preview 60: 7216

Brand new iPad 2018: 16653

Moto G4 phone Chrome: 37735

Windows 10 Laptop:

  Firefox Nightly: 2278
  Chrome Canary: 4907
  Edge: 16332

Have you tried enabling the experimental new WebAssembly baseline compiler in Chrome Canary?

As a matter of fact, I just did! Still getting crushed at 4546

Check with --js-flags="--no-untrusted-code-mitigations", it should be about 10-13% faster. https://github.com/v8/v8/wiki/Untrusted-code-mitigations

This does not sound safe.

Aw, Snapped! on me

Pretty soon we will be running Chrome on Firefox.

https://www.destroyallsoftware.com/talks/the-birth-and-death...

IIRC they've also always had the fastest Typed Arrays. I wonder if there is a connection there - it wouldn't surprise me if memory access is the biggest bottleneck for WASM in other browsers.

A big part of the score is compilation time, which improves a lot with Liftoff, our baseline compiler for WebAssembly in V8. We've been in contact with the benchmark authors and have identified other bottlenecks in our engine (e.g. instantiation time), which we are now working on optimizing. Look forward to some increasing scores soon :)

Liftoff is still substantially slower at initial compilation than Firefox. Are there any plans to close the gap? A Unity exported game takes over 40 seconds for a new model low end Chromebook to compile. The upcoming 3 fold decrease will be nice, but FF could do it in 2 seconds on the same hardware.

https://lukewagner.github.io/test-tanks-compile-time/

FF 61: 78.4 MB/s

Chrome Canary: 9.7 MB/s

Chrome 67: 3.3 MB/s

We still have a bottleneck at instantation time where we compile many JS<->WASM wrappers. We are working on parallelizing this compilation as well, so we'll get another bump out of that.

Samsung Galaxy S9+ (w/ Exynos):

  Firefox: 6900
  Firefox Nightly: 7874
  Google Chrome: 10602

On my phone (OP6) FireFox beats Chrome by two whole seconds

OR I can just read and edit PDFs in the peak-optimized, ahead-of-time-compiled native PDF reader that comes on my computer.

Ha! Not sure if you've seen it, but we offer native SDKs for iOS, Android, Windows, macOS and Linux. Our Web SDK also has a version that uses PSPDFKit Server via a Docker container to get faster performance.

The performance difference to native is smaller than you might think though - see our post from a few months back: https://pspdfkit.com/blog/2017/webassembly-a-new-hope/

PSPDFKit for Web Standalone is a great choice for an easy-to-deploy web viewer, annotation editor and form filler, and performance is great. Definitely better than pdf.js - which is typically where people start with, before they upgrade to PSPDFKit for Web.

> Our Web SDK also has a version that uses PSPDFKit Server via a Docker container to get faster performance.

...what. why would I want that over a library that I can do whatever I want with?

Right? The built in safari reader does everything correctly and interacts with the rest of the OS properly. I assume that the built in chrome one also Just Works (tm).

Also why would I want to download an entire pdf engine when i already have an AOT version? (Stealing that phrasing by the way :) )

Safari's PDF viewer doesn't allow you to annotate, especially not in real time with other users: https://pspdfkit.com/instant/

It's also quite basic when it comes to forms and often loses data.

For cases where more performance is required we always recommend using PSPDFKit Server, so PDFs render in native code on a high-power machine, and very likely using cached tiles.

>Safari's PDF viewer doesn't allow you to annotate, especially not in real time with other users: https://pspdfkit.com/instant/*
I never asked for that though, so there's that.

I can see people wanting that, but then I'd have to wonder why they are using PDFs for that purpose and not something like Google Docs.

Think construction plans, PowerPoint presentations, contracts - there are many documents that don't fit the classical Word document model.

I wonder if the PDF file format is the best one for such kind of content

Turns out that it is - it's amazingly flexible and pretty much anything out there will be able to render it. There's practically no other formats that can claim that.

But why would one expect a browser to support that feature for PDFs?

You wouldn't, and it doesn't. But webapps and websites can usefully do it.

On my computer, the native PDF reader is xpdf, which is made of security holes, or something based on libpoppler, which is made of patches to xpdf.

You mean "open cell security foam" of course. It means the same thing but sounds better for marketing.

I prefer the sandboxed PDF reader in my browser to the native one that's not sandboxed
https://www.theregister.co.uk/2016/10/24/apple_security_upda...

FYI: WebAssembly is effectively starting to get WebAssembly only APIs. Chrome is shipping the WebGL commit API which is designed to support spin loops in native apps (others are expected to ship it too)
Most native OpenGL apps run in a spin loop as in
while(true) { update(); render(); glSwapBuffers(); }
To support that model TPTB added a worker only `gl.commit` function that does effectively the same thing as `glSwapBuffers` so that native apps can keep their spin loops and don't have to refactor to be event based.
Of course if you decide to use a `gl.commit` spin loop your worker can no longer receive events (no onmesagge, no XHR, no websockets, no fetch) so you're supposed to use SharedArrayBuffers to communicate with a spin loop worker.
SharedArrayBuffers are being re-enabled once they are isolated to same domain only processes.
AFAICT this is the first browser API that is really targeted exclusively at WebAssembly. I'm guessing more will come. Not entirely sure how I feel about that but I guess it's inevitable.

Are you sure gl.commit is WebAssembly only? That doesn’t seem right to me, but I could be wrong: https://hacks.mozilla.org/2016/01/webgl-off-the-main-thread/
What is TPTB?
SIMD is another WebAssembly only API, hopefully coming soon.

"the powers that be"

By WebAssembly only I don't mean you can't call it from JS. I mean rather it's designed for Wasm not for JS. JS is generally event oriented. Native C/C++ games are usually spinloop+polling oriented. With the current design of commit is for spinloops but once you do a spinloop you can no longer receive events.
That blog post you linked to is 2 years old and out of date with what browser are actually planning on shipping.
For JS you no longer need commit at all. canvas (both 2d and webgl) work offsrcreen just like they do without offscreen. You render, when your event exits the changes get magically propagated to the corresponding canvas. commit is solely for when your event never exits (you're in a spinloop)

> The team was especially helpful in pointing out bottlenecks in our JavaScript implementation, and we’re already incorporating these changes into our next PSPDFKit for Web release.
Any chance you can talk about those bottlenecks? Might be interesting.

We should be at the point where network computers are practical for casual users. A full install of the BeOS operating system is just 45MB. A tricked-out one was barely more than a couple of hundred MB. We should now be able to have operating systems which are mostly cached from online connections, running in clients like Web Browsers, but purpose built for running such an OS.

What's the difference to benchmarks like https://hacks.mozilla.org/2018/01/oxidizing-source-maps-with...? The results seem vastly different.

One is a large c++ compiled ti asm.js and wasm. The other is wasm compiled rust vs javascript. How are they comparable?

GP is just wondering why the results are so different.

You can't compare small/micro-benchmarks that call into WebAssembly and then get the result with a larger full-stack application where parts are in JS and parts in WASM - both are highly interlinked. This is much more reflective of real products. We couldn't find a benchmark that shows such end-to-end usage, so we built one.

the standalone version produces a dom tree?
I expected canvas rendering, which would be MUCH faster. (?)
<div style="top: 628.75px; left: 125px; position: absolute; transform-origin: 0px 0px 0px;"><label style="width: 38px; height: 38px;" class="PSPDFKit-43pb73xrbckjuggksebahxjw9 PSPDFKit-Annotation-Widget PSPDFKit-Annotation-Widget-CheckBox"><input name="Checkboxes 2" value="Checkboxes" type="checkbox"><span class="PSPDFKit-v4b9qprfhz2hybx1p43s4xgx6 PSPDFKit-8vpzte5r65p8pf9q4d61qcb34u" style="border-width: 1.25px;"></span></label></div>

If you render to canvas, you would need to reimplement scrolling and zooming, checkboxes and input elements. Any custom reimplementation of this stuff will end up slower and clunkier than the optimised, native versions already in browsers.
Also (as far as I know) canvas content is inaccessible to screen readers. Accessibility matters.

No problem.
http://example.qt.io/qt-webassembly/quickcontrols2/gallery/g...

> Browser vendors can reach out to us and obtain a more permissive license key so that the benchmark can run on different machines and even on their continuous integration servers.
Sounds like a great idea for encouraging performance improvements and avoiding regressions!

wondering why those benchmarks weren't run on Linux distros like Ubuntu, Debian, etc.

Now all we need is to expose a graphics context to wasm. No need for DOM support.

You already get 2D, WebGL, WebGL 2 through canvas

Great work, looks like a very useful PDF toolkit. I've been working on a side project implementing something similar for some time but has stalled due to life getting in the way. Can you share anything publicly in terms of pricing?
I see everything is through a sales person now and I'd like to get a better high level understanding of high level costs for passion projects (<1000 users), small-medium (<100k), and enterprise (1M+). What metrics are you charging usage on?

Metrics are required platforms (iOS, Android, Web, Windows, macOS, Linux), components (see matrix on https://pspdfkit.com/pdf-sdk/), number of users, number of server nodes (if Docker-based), number of apps/[sub]domains, internal/external access and expected support, to name a few. We're usually quite fast to figure something out that works for both parties, pricing is in the 4 (passion projects) to 6 (enterprise) figure range. Simply drop us a message via https://pspdfkit.com/sales to get things started. We try to make this process as easy as possible.

You should probably consider putting something basically exactly like this paragraph on your site, especially the pricing range.

^ this. Please provide a pricing page with a few options and even a usage calculator. Without it, you'll be hard pressed to find enough folks giving this a real shot and will just assume that it's enterprise-only and out of their range.

Safari is so slow

Looks like they are working on it, which is good news ->
From https://twitter.com/steipete/status/1013035117076467713 :
Shoutout to the Safari team! They were able to track down the slowness in the current WebAssembly implementation and ALREADY FIXED IT. Patch is here: https://trac.webkit.org/changeset/233378/webkit
(trace points caused a significant (4x!) slowdown as our benchmark performs many calls into the VM)

One thing that would be super nice, in theory, would be compiling WASM modules on-the-fly from the text format.
In a sense, it would be like a JIT for the Web, and one possibility for bringing existing JIT-based languages to WASM.

In case anyone else was wondering, you actually CAN do this and JIT WASM in your browser via Binaryen:
https://github.com/WebAssembly/binaryen/blob/master/test/bin...

What is the most performant compile to wasm language? Or are they all similar?

It's really hard to say, especially as the wasm interpreter stuff itself is under so much flux; different implementations act differently, etc.
There's also the question of what aspect of performance is of interest to you. Do you include the download time in performance? If so, languages that produce leaner wasm files will be better performing, even if the execution is identical. Do you only care about execution time? Then that aspect isn't relevant. etc.

Since all languages should be able to compile to wasm, that question is pretty close to "Which language is the most performant?", which is a can of worms that I'm not going to open. Maybe something like the Benchmarks Game would be useful here. Or already exists, and someone could point me to it :)
A similar question would be "What languages do you know of that are focused on performance and compiling to wasm?" For that, I'd say C++, Rust, and Go, though I'm sure there are others.

That depends on (a) how you define your measure for "performance" (see other comments that talk about the difference between microbenchmarking and application benchmarking), (b) the compiler that you use, (c) the WebAssembly implementation that the code finally runs on.
So, do not expect a clear answer to that questions. Probably no one really knows because of a lack of benchmarks and deployments in the wild that have been analyzed. That's why they made the benchmark.

If you really do mean fastest and nothing else, you can't really beat WAT, the WebAssembly text format.

> In the future, Microsoft wants to bring more features — which are currently not possible in asm.js (or JavaScript) — to WebAssembly.
This is very vague, but already disturbing

In my understanding, SIMD is an example of a feature like this, as is threading. It’s not just Microsoft that wants this.

I'm guessing these "features" will be tied to the Microsoft Store, something they're already doing with PWAs.
Many were quick to criticize Valve and Epic about their own criticism of the Microsoft Store, which is why Microsoft is now using a "boiling the frog" strategy to tie everything to the Store over the next decade or so. And this process will likely accelerate once Windows 7 is dead and buried.

For those that don't know it, WinRT always supported hosted web apps.
Microsoft has picked up on Google's PWA ideas and is migrating UWP hosted web apps to it.
I expect Google to follow a similar path with Android and ChromeOS.

>This is very vague, but already disturbing
Yeah, god forbid the platforms gets more powerful...

Maybe they'll be the first to implement tail calls!

- Firefox 62b5: 1429
- Chromium 68: 5722
- Safari 11.1.1: 6500
On 2017 Macbook pro running macOS 12.13.5 on battery power

I tried on Ubuntu 18.04. Firefox around 2k and Chrome around 10k

Reply on news.ycombinator.com