I have to say that, while I haven't looked into the details, this is quite impressive. It runs really smooth on my not-so-great desktop. A good sign that Javascript is no longer the great bottleneck it used to be.
The main performance hog is not the decoding, but the conversion of the decoded YUV image to RGB for the canvas element. I believe this step could also be done on the GPU with WebGL.