A Simple Explanation: VLC.js | Hacker News Reader

A Simple Explanation: VLC.js

ascii.textfiles.com

135 pointsvmorgulis66 comments

At what point will the current trend to treat the browser and WebAssembly as a compile target for traditionally fat client applications raise the need for a browser package manager ala Apt or Yum? When will dynamic linking become a necessity because due to memory usage, disk space, and patches for 0day exploits in upstream libraries?

It seems like the more the browser takes on the role of an OS the more it will need all the stuff that OS's have built up over the years to make it manageable.

And the strange thing is: I can already start a VLC binary over the network via my file manager.

A lot of people think we need to start rethinking the web. But as the poster points out: "but that’s what we have and here we go"

Yeah at some point we by default decided the benefits of just go to this URL as a way to install software outweighed the negatives. It wasn't even really a deliberate decision collectively. We solved problems in a locally optimal way but I suspect it's not the globally optimal solution.

That is kind of the role filled by "npm" and "yarn". They are obviously a lot different from OS package managers, but ultimately they solve a parallel type of problem for the web.

Actually no. Those tools are development package managers not user package managers. Two very different use cases. A development package managers is part of how the developer builds the application. A user package manager is how the consumer of the application installs it while also managing all the shared libraries and installing security updates for each of the pieces.

You are absolutely right in a way. I guess my real point is that a "user package manager" isn't likely to be a thing that ever makes sense on the web.

The whole point of most of these projects I think is to avoid having to install anything. You want something to just show up without any user interaction. I think for the most part everything is just going to be statically linked.

We have (or had) that. Chrome is removing web app support (the Apps button on your favorites bar). Firefox removed their app support after FxOS died.

Nobody wanted these things. Users didn't understand the idea. The APIs were clumsy. The framing of the apps added little value over a traditional web app. Portability was a joke.

The best thing to happen to the web is service workers and the new world of "offline" APIs, like the push notification API. They allow pages to get the goodness of a desktop app without the downsides of installing a "native" app.

(source: I helped build the Firefox Marketplace)

Those are more of an app store and less of a package manager. Apt and Yum solve a problem users don't know they have. The app stores solve the problem of obtaining and installing the application. They don't solve the problem of managing shared libraries and quickly updating those shared libraries.

For an application like VLC this sort of thing become really important. More important than your typical email client or document editor.

On the web though, you don't need the user to manage libraries. In the same way you don't install a web page, libraries are simple script tags pointing at URLs. Pages will never reference incompatible versions. Browser caching makes "sharing" of libraries transparent. What's there left to manage?

Is this still true for an application like VLC compiling to WebAssembly that uses other downstream libraries?

What if VLC uses downstream libx and Audacity also uses downstream libx and downstream libx has a critical 0day. I'm not super clear yet on how the WebAssembly build pipelines works. Is dynamic linking done through script tags? or are the binary blobs statically compiled? I'm genuinely curious here how this is all supposed to work. How does one ensure that downstream libx is updated for all of the WebAssembly applications that use them.

> I firmly believe that this project, fundamentally, would change the relationship of audio/video to the web.

Aren't they banging open doors? HTML5 video and audio is great and with WebAudio and WebGL you can manipulate every aspect of the experience. EQ, filters are possible. What else do you need? Subtitles?

I just don't understand what problem are they trying to solve because "I can't run VLC in the browser" certainly isn't one.

You can even get subtitles on top of HTML5 videos with simple positioned elements with text shadows for borders. I can't say I understand the motive of the project either, other than "see, everything can be done in JavaScript!"

Or just use a `<track kind="subtitles">` and reference a vtt file?

Which is a format you need to convert to rather than use existing srt or anything else really.

Which doesn't allow SRT subtitles, or XML subtitles (hello German public broadcasters!), or ASS subtitles (hello anime fandub groups!)

Doing this natively in JS via canvas allows you exactly that - rendering these things natively, in real time, with all the support for rare features you could ever need.

Much harder than it sounds if you care about live streaming, video playlists, and cross-browser support. Don't forget to mention the 3 big subtitles formats (608/708/VTT) and some smaller ones (SRT/DFXP).

Also, here's a fun quirk of the subtitles track element: you cannot remove a track once added.

Another obstacle for full FCC compliance: styling. You must allow users to style your captions. Captions are put in the shadow DOM - Firefox does not allow you to style the shadow dom.

Don't forget to add IE into the mix!

Support for the same wide variety of formats that VLC now supports.

The author works for the Internet Archive and it obviously has a huge amount of files in a variety of formats that are not playable natively by browsers, and probably never will be.

VLC's support comes from its support for the codecs. Why not port the codecs to JavaScript so that the existing HTML5 features can make use of those formats?

For this to work, the codecs need to be able to compile to JS via emscripten, so what you are asking for is necessarily part of the task.

But playing a DVD, for example, isn't just about codecs, they have menus and various other features that need to be implemented too.

I'm curious how VLC usually draws inline UI elements, menu effects and such. If they need to be implemented separately per platform or if they just need to create an adapter for drawing the usual VLC UI.

I think they just use Qt.

Supported on tons of platforms, makes sense I guess. Linux has some variant for GTK as well if I recall correctly.

"I can't play back media in the browser that it doesn't natively support unless I transcode it first."

Remember that this is the Internet Archive - they have huge quantities of content that they're seeking to make more accessible by making it one-click playable in the browser. It's worked out brilliantly for disk images from the 8-bit computer era, and I expect this new project will work out even more so.

I think its mostly interesting for supporting niche file formats directly, as he says:

"And completing this thought, by choosing something like VLC, with its top-down open source condition and universal approach, the “closing of the loop” from VLC being available in all browsers instantly will ideally cause people to find the time to improve and add formats that otherwise wouldn’t experience such advocacy. Images into Apple II floppy disk image? Oscilloscope captures? Morse code evaluation? Slow Scan Television? If those items have a future, it’s probably in VLC and it’s much more likely if the web uses a VLC that just appears in the browser, no fuss or muss."

This makes a lot of sense for the internet archive. There are many "lets do it in js because we can" projects, this is not one of those.

He explains near the bottom. Rather than convert weird old formats to something the browser understands, which is what they do currently, this lets them play them directly.

He hopes having that capability in the browser will inspire people to add even more obscure formats to VLC. So it's a virtous cycle.

> Rather than convert weird old formats

In theory it would also enable browsers to support 'wierd' new formats like BPG [1], FLIF [2] and Lepton [3] which are never going to get widespread support or any support at all.

[1] https://en.wikipedia.org/wiki/Better_Portable_Graphics

[2] https://en.wikipedia.org/wiki/Free_Lossless_Image_Format

[3] https://blogs.dropbox.com/tech/2016/07/lepton-image-compress...

It seems to me that they would use these HTML5 features and APIs to implement audio and the actual pixel output from the VLC codecs.

But this would mean support for an incredible number of input formats.

At least that is how I understand their goals.

This seems right: just perform the conversion in the browser and play the output in a <video> tag. The alternative would be to draw frame-by-frame in a canvas which would be very difficult.

Ah, you mean transcode to something the browser actually handles? I think that transcoding might be many variants of terribly slow though. I'm not sure drawing it to canvas frame by frame would be all that hard, VLC already does some kind of frame-by-frame drawing I'm guessing, on the output end. Canvas should be plenty fast enough as long as the decoder can keep up I imagine. I'm thinking this might be a good use of Web Workers to offload decoding to another core to avoid hanging the main thread (UI).

I'm mostly guessing.

I wonder how heavy different encodings/formats will be to decode in JS. Haven't worked with video on that level of detail, but I imagine that there a plenty of optimizations that can be done in low-level languages to get solid performance that JS might not deliver.

The Dash.js MPEG-DASH reference player can handle EBU-TT-D subtitles.

Correct subtitles are big deal for i18n and accessibility viewpoints.

My goal/wish would be that it could playback local DVD's somehow... :)

VLC also might be "more robust" (in terms of accomodating for file corruption) than your browsers' player, might provide a more "stable" experience cross browsers, or allow things like "convert that video file for me in the browser to images and show me those instead" etc. FWIW.

From the previous discussion:

vespakoen noted that there is already ffmpeg library compiled with emscripten:

http://bgrins.github.io/videoconverter.js/demo

My speed test of webm to mp4 conversion (it doesn't display the video, it's just ffmpeg) showed the JS version around 10 times slower than the native ffmpeg.

The test was on relatively recent i7 and the latest Firefox (certainly having as fast asm.js as possible) then (around a month ago). I haven't investigated the cause for such significant speed difference.

I have a project that decodes mpegts packets and renders the frames to a webGL canvas using FFMpeg. https://github.com/colek42/streamingDemo. Full disclosure much of the client code is copy/pasta from https://github.com/opensensorhub

Edit: add attribution

is there a live demo some/anywhere?

Sorry, I am using it to proxy UDP streams which requires a backend component due to browser restrictions. If I hosted a demo it may end up costing a nontrivial amount of money. I am developing this closed source for my employer, but if there is enough interest I can push some of the changes to the open source repo.`

Power consumption and performance are big concerns, because JavaScript (software) decoders can't leverage hardware acceleration.

But archive.org wants to preserve and publish many formats that are not high resolution, high frame-rate video, and we can expect both performance and power consumption to improve over time.

HTML5 canvas can use hardware acceleration, of course. If the codec being used is suitable for HW acceleration (i.e. h/x624 yes, VP9 no - generally)

But wouldn't a browser plugin be easier to develop and also easier to use for the end user? Exactly what is gained by having it run in js? (Better portability maybe? But VLC already runs on almost all platforms natively...?)

Because the user doesn't have to install anything.

>But wouldn't a browser plugin be easier to develop and also easier to use for the end user?

The trend for the last couple of years or so has been for browsers to deprecate third-party browser plugins.

Because every browser has different plugins and their concept tends to change every few years, for good (security) or bad (politics) reasons. Once VLC.js runs, you can reasonably expect it to run forever and be just another compilation target.

Exactly.

For example, Mozilla is deprecating NPAPI just as fast as it can, because it long predates e10s (Chrome-style process isolation) and they do not play well together. e10s has been so long delayed already, and Firefox's performance problems are so intractable to solve without it, that from the perspective of a very long-time Firefox user like me this is very obviously the correct course of action.

This deprecation will break the existing VLC plugin for Firefox, but that's not a huge deal because it never has worked very well in any case - developing a reliable NPAPI plugin is a seriously hard problem in general and has been since browsers got complicated enough to do interesting things in the first place. You also have to restart the browser to make any changes to the set of installed NPAPI plugins, and that's a lot more of a problem than it was before the browser became an application runtime. Now that the JS runtime is fast and capable enough to run "real application" code, we derive a much better experience by leveraging that power than we do by targeting a legacy interface that is relatively ill documented and much easier to get wrong in ways that break the whole browser rather than just one page or tab. We also get cross-browser compatibility of a sort that NPAPI never could offer.

I get that a lot of people think it's weird for the web browser's page scripting language, which was nothing more than a cute toy a couple decades ago, to be eating the world like this. I think it's weird, and it's how I earn a living! But, weird or not, it's also the best application platform we have or likely soon will have for a broad and widening variety of use cases, and that is because, while no more perfect than anything else, it actually works really well. Railing bitterly against technology that works really well seems potentially counterproductive to me. But I appreciate there exists a diversity of opinion on the subject.

> Once VLC.js runs, you can reasonably expect it to run forever

You can reasonably expect it to run... until the first non-backward compatible change in anything the code depends on. The major assumption for this to keep working is that everything has to be permanently kept up-to-date for all the breaking dependencies. And I assume it will always be easier to produce a Linux or Windows binary that works relatively long without changes than anything that non-trivially depends on the web technologies.

A few years ago, as far as I know, it was still possible to run the Netscape Navigator binary which was around 20 years old on the recent Linux kernel.

Killing the flash player plugin for video with another plugin?

Security is one good reason: nearly all native media players (including VLC[1]) are full of serious security vulnerabilities that let people take over your computer via crafted video files.

[1] http://www.cvedetails.com/product/9978/Videolan-Vlc-Media-Pl...

Does this mean that developers will go as far as maintaining a port of FFMPEG to JavaScript? Being able to decode 10bit videos in pure JS sounds both insane and insanely cool

> a port of FFMPEG to JavaScript

It already exists, see my other comment for more details.

There are a number of ports powered by Emscripten (https://github.com/Kagami/ffmpeg.js)

I wrote about some stuff here https://paul.kinlan.me/ffmpeg-ideas/ when building https://paulkinlan.github.io/deviceframe.es/ (I do a lot of screencasts and need a simple way to scale this out to other people that doesn't involve installing ffmpeg and scripting)

EDIT and Note: Also I just saw your comment about videoconvert.js

Do I understand it correctly that you want the users to do javascript ffmpeg processing on their android devices? I admit I have no idea what deviceframe is, so I'd also like to read some explanation on that too.

Have you compared the speed of Android javascript ffmpeg vs PC native? What happens with the battery?

Javascript and as soon as possible WebAssembly, as I understand it.

I think mobile is a secondary consideration in this at this point. Considering it is connected to the internet archival efforts and simply ensuring the survival of numerous video/audio formats I imagine they are fine if it isn't efficient but simply effective at this point.

Edit: on second thought, you might not be commenting on the VLC.js thing, I haven't read about the ffmpeg js stuff ;)

Sadly, that answer is a completely missed conversation topic.

I've asked about

https://paulkinlan.github.io/deviceframe.es/

do any of these ports allow playback of a video "in the browser" by chance?

This is one of the reason I'm truly excited about the future of javascript in its ES6 or Typescript form.

If they manage to make this run and use emscripten and manage to build all the video computation optimizations.. how far of a stretch is it really to combine lodash+emscripten+BLAS/LAPACK to build a high performance computing platform based on javascript ?

Honest question: Why is building an HPC platform on a language designed in less than 2 weeks something to aspire to?

Why not aspire to replace javascript entirely with something better?

huh?

ES6/emscripten/Typescript was not designed in 2 weeks. could you talk about what you mean.

He's talking about the first JS specification which was done in 2 weeks IIRC. I've seen such comments several times which completely discard all the effort made to improve JS until now.

He's talking about Javascript itself. The language has a ton of short-comings, hence the rise of languages that transpile to JS. I agree with the previous sentiment, we should replace JS with something better. JS is a nightmare.

The first version of the language, upon which all other versions have been based, was done in 2 weeks. There are fundamental shortcomings in JS that exist to this day because of that, and because the web has to remain backwards compatibility. So, despite all the improvements over the year, JS is still poor in some areas, particularly when you're talking about porting applications to JS. It was never designed for that.

...because the goal of the Internet Archive is to get playable versions of archived media into your hands, and Chrome/FF/Safari/Edge/etc represent a powerful & free, execution target that everybody already has.

Besides -- and this is key -- they're not rewriting it in Javascript. Javascript (via Emscripten) is a compile target for the code they already have.

Those who have read the article know this; it's right there in the article:

    I know that it sounds like porting things to Javascript 
    is crazy. I find that people think we’re rewriting things 
    from scratch, instead of using Emscripten, which compiles 
    out to Javascript as a target (and later WebAssembly). We 
    do not write from scratch.

> how far of a stretch is it really to combine lodash+emscripten+BLAS/LAPACK to build a high performance computing platform based on javascript ?

You almost make me wish Java applets or sigh ActiveX back.

Why would you want to do that? R, Python and Julia already exist for using a dynamic language for HPC scientific and statistical computing, and they can be run inside a browser using something like Jupyter.

Those languages already have the libraries, tools, community, etc.

Wouldn't webxmp (also emscripten based) be a better match for MOD/S3M/XM support:

http://www.wothke.ch/webxmp/

That's really neat! I still have a few music modules I can't part with.

This is very exciting and to a lot of people, doing things with only a browser is so much enabling!

Reply on news.ycombinator.com