HNHacker News
TopNewBestAskShowJobs

padenot

2,536 karma · joined June 21, 2011

Media developer at Mozilla, working on WebRTC, <audio> and <video>, Web audio.
submissionscomments
padenot··on Firefox tooltip bug fixed after 22 years
https://firefox-source-docs.mozilla.org/contributing/stack_q.... It's then two clicks to diff two arbitrary versions of a particular patch. For complex patch sets that take a few rounds of review, it's nice to understand what changed since last time you looked at it. When reviewing code from e.g. a beginner, it helps to quickly check that all comments have been addressed and nothing else has changed, etc.

Then you merge ("land") the patch stack in one unit, and it's tested in one unit, and bisection tooling knows that it is the case and it's nice.

padenot··on Firefox tooltip bug fixed after 22 years
I use GitHub everyday, and problems, or what I miss from the mozilla stuff are:

- git is not as modern as mercurial (no absorb, no evolve, no nice ncurse gui, weird explicit branching model). I've been using both daily for more than ten years, but mercurial is largely superior in every respect

- no way to properly do patch stacks and proper reviews, with static analysis and linting integrated into the review tool, no way to lose comments by force push or that kind of thing (phabricator is amazing)

- no way to properly do private issues for security bugs or touchy audits that could reveal large problems

- no hierarchical categories for issues, no severity, priority, little provision for advanced filtering for a project our size. Bugzilla is very nice and much better than anything else I've tried. You can use tags for a lot of that on gh but it's messy - code browsing is primitive compared to https://searchfox.org/ (but most code browsing tool are, in comparison)

- my notifications are completely flooded by lots of useless information on GitHub, but that might be fixable

- our CI system (treeherder/taskcluster) scales, works on Linux/Mac/windows/Android and a bunch of version and arch, integrated with all of the other tools mentioned. Things such as auto-running tests based on the content of the patch, automatic categorization and prioritization of intermittent test failures, or auto-recording test failures and offering a pernosco recording showing the issue are just some of the features that we use daily without even thinking

There's probably a lot of other stuff that is better, but that's a nice start. If GitHub had a real patch stack management story, a real issue tracker and a real review tool it would go a long way, but that's not the case.

padenot··on Firefox tooltip bug fixed after 22 years
It's one command to clone from git or hg at your convenience, one line to install the development dependencies (that figures out the os/distro and does the right thing) and one line to build.

Then one line to install the command line tooling to submit patches and another line to submit a patch (or a patch series).

All the docs are there for all major platforms, aimed at first timer in our code base: https://firefox-source-docs.mozilla.org/contributing/contrib...

It used to be much more complex than that but this isn't the case anymore. Using GitHub (that we need to use anyways because lots of projects and lots of web standards use it) feels like banging two rocks together compared to what our tooling can do, especially at that scale.

Source: I work at Mozilla and frequently onboard contributors of all backgrounds and levels.

padenot··on Finding and exploiting vulnerabilities in H.264 decoders [pdf]
The mp4 demuxer is indeed in Rust [0], and runs in the content process (= the process in which the web page is loaded).

We don't have a h264 decoder in our source tree, we use the platform's decoder (because of patents). It's very often in a separate, dedicated process, and when it's not, it's in the GPU process, because when hardware accelerated decoders are used, they're using more or less the same resources as the rendering code.

Those other processes with the tightest sandbox possible (per process type, per platform, etc.), and don't have access to the web page.

On Linux, the platform decoder we're using is `libavcodec` from FFmpeg, but that's still in a separate process with a tight sandbox.

We're also doing something interesting, which is compiling libraries to WASM and then back to native code to get memory safety [1]. This is used when performance isn't critical (unlike codecs, so, e.g. a demuxer that we don't want to rewrite in Rust).

[0]: https://github.com/mozilla/mp4parse-rust/ [1]: https://hacks.mozilla.org/2021/12/webassembly-and-back-again...

padenot··on FFmpeg 6.0
Firefox vendors a subset of ffmpeg, to decode royalty free codecs such as vp8/9, mp3, flac, on all OSes. We're using our own decoders/demuxers for other royalty free codecs for historical reasons.

It can also use the system's ffmpeg to decode patented formats such as h264 or AAC, on desktop Linux, if the copy of ffmpeg present on the host includes non-free codecs (same on other OSes, with their respective system libraries).

padenot··on Save web pages as PDF in Firefox for Android
This is in the works. It's the same codebase as the desktop version, which is written in js.

I've tested it on low end Android phones from 2017, it works well, when compared to the native offering, and also on modern high end phones, it also works well of course.

What's missing is mobile specific UX, such as pinch-to-zoom, and UX/UI refinements, so it's nice to use.

It has form support, annotations and other goodies like the desktop version as well, which I find very useful.

about:config in Firefox for Android Nightly,pdfjs.disabled to false to give it a go.

padenot··on Show HN: A 64-step drum sequencer I built using React and Redux
I'm not sure how many Windows users RtAudio has, and the kind of machines people are using with it, and their expectations, but in Firefox, the answer is hundreds of millions and "any kind of machine that has existed", such as single or dual-core machines that are running dozens of modern web apps in parallel, so load is high. That might well explain why it was reported so quickly. It's quite noticeable with a standard load testing procedures (e.g. "how many parallel oscillators can this computer/OS/driver do at this latency and be glitch-less under load").

It is acceptable to do audio work off the main thread (as long as buffering is in place to prevent glitches from scheduling delays, and it's prioritized appropriately), but maybe you mean "on the main thread". In this case, it's not really recommended, but sometimes, it's what you need to do, for example when emulating an old console where everything is synchronous in the render loop.

Locks and allocations are best avoided, but modern locks and allocators are amazing. If the lock is _not contended_, it's very fast, and it's fine. If the allocation comes from a thread-local pool (some allocators do that), it's also very fast and fine, but that needs to be true across OSes, etc. for us. In other case, which is most of the time, it's best to avoid those.

We're not allocation nor lock-free entirely in Firefox though, but it's been thoroughly profiled - when know what we'll optimize when it's time to do so, but for now it's fine under extremely heavy load in a normal situation (we have locks when the audio devices are unplugged for an example of a "non-normal" situation). This is largely due to legacy (the Web Audio API, for example, is about 10 years old).

padenot··on Show HN: A 64-step drum sequencer I built using React and Redux
I work on the Web Audio API at Mozilla, here is an article that I wrote that explains what goes wrong here, distilled to the core: implementing a metronome in literate programming in a way that will _never_ glitch: https://blog.paul.cx/post/metronome/. The result is nowhere as interesting as this app, but this is to get the point across.

https://www.html5rocks.com/en/tutorials/audio/scheduling/ is an article written by my former fellow co-editor of the Web Audio API specification, that talks in length about this.

In 2021, with the AudioWorklet [0] being available in all major browsers (including Safari as of a few weeks ago!), arbitrary computations can happen on the real-time audio thread. It is now possible to send a copy of the sequencer data (which note when, what sample, what velocity, etc.) to the real-time thread, and let it play, unaffected by the load of the machine, be it from the browser or other programs (until everything breaks down because you're also overloading the audio thread).

I think this app is very good and does the best it can under the programming model it's using, considering the environment it's being run in, but if more robustness is needed, there is now a way !

On your last paragraph, it is still _absolutely_ critical. An audio software that doesn't use a real-time thread for rendering cannot render audio without glitches, even under no load, even on a fast machine. You don't need to tune the OS [1] and pick special hardware these days: today's computers (and even computers from 10 years ago or older) are perfectly capable or running reasonably complex audio processing workloads _if the software is written correctly_. This is regardless of the use case. When we disabled the real-time thread on Windows by mistake in Firefox (adjusting priority of the process level, and then that was inherited to all threads, very quickly reverted), we had reports in a matter of _days_ (if not the next day), and the Firefox Nightly population is not that big. This was I think playing regular videos on social media, not even anything complicated.

If the author of the app, or anybody, wants to profile the app with the right tools, I've written a blog post [2] about this as well. I hear this tooling is used by audio professionals, now, and that they're happy with it. Happy to hear about any suggestion though :-).

For your Firefox-specific question, you're right that increasing the content process count will make everything better, and we're constantly improving memory usage so that it's possible to do so even on lower-specs machines, maybe it's worth a try. Please look at the "resident memory" values, and not, say, virtual or shared, because we make heavy use of shared memory and copy on write facilities of the OS to limit the memory usage when using multiple process (which we need to do for origin isolation and avoiding Spectre issues and sandboxing various bits of the browser as hard as we can, for example).

[0]: https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkle...

[1]: (but you can to improve robustness further, and it helps to have good drivers, such as Jack on Linux, WASAPI with MMCSS on Windows or ASIO, or just the normal stuff on macOS)

[2]: https://blog.paul.cx/post/profiling-firefox-real-time-media-...

padenot··on Permission.site
Can you give us a couple of these crashes, that you can find in about:crashes (with the date sometimes its easy to find the right one) ?

If you send a link to padenot@mozilla.com I can have a look, I'm on the media team at Mozilla (that includes WebRTC).

Thanks!

padenot··on Porting Firefox to Apple Silicon
The above is correct. The CDM is very heavily sandboxed, a signature is used, and therefore it can't really do anything apart from what it's supposed to do (which is very little, taking encoded data, a key, decode media).

Source: I'm on that team, but I don't work directly on this.

padenot··on How Vivaldi browser is different from Google Chrome
Firefox audio developer here, would you mind giving me a link to a couple of those "spatial audio tools" that don't work in Firefox?
padenot··on Firefox 84.0
Would you mind capturing a profile? https://blog.paul.cx/post/profiling-firefox-media-workloads/... explains how.

It's very very very simple and quick, there is a video demo and a text explaining what to do, and then you can open a bug at https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&comp..., with the profile link (you can login with a github account or email). This will be triaged and hopefully we'll understand what's up. Please use a Nightly build or a Mozilla-provided build when doing so (so there are symbols). It's just a matter of downloading and untaring the tarball.

Please also attach a copy of `about:support` raw data (button at the top), so we can understand what your configuration is: desktop Linux systems can be pretty diverse and sometimes it isn't clear what works, what doesn't and why.

If you're playing 480p, then youtube might be serving you AV1, and this is (for now) using the CPU (trading a lot less bandwidth for increased CPU usage), you can check the codec it's using by right clicking on the video and clicking on `Stats for nerds`, there is the codec string. You can adjust your codec preferences in your youtube profile I believe.

padenot··on A pure WebAssembly / JavaScript port of FFmpeg
Firefox media developer here.

It's a bit hard to understand what the problem is, sorry.

You can block video and sound auto-play altogether, or just audio, or none, in `about:preferences#privacy`, and choose or check exceptions you've granted, but if I read the grand-parent comment correctly, an appropriate setting has already been put in place.

Additionally, if the website really wants to play audio against a visitor's will, it can try to hijack clicks and touch events. There are some plans to tighten this, the spec was designed to allow implementing various policies. It should also be possible to write a content blocker for this as a Web Extension (maybe it already exists).

Does this happen on all websites? On some websites, that you could possibly share with us, so we can have a look and understand what's up ?

Can anybody that is not satisfied about this open a ticket at https://mzl.la/363s8KN, and put in `:padenot` in the box `Request information from` (log in with github or a bugzilla account) with some info about the websites ? An alternative would be to send me an email, username at mozilla.com.

padenot··on AudioMass – free, open source, web-based Audio and Waveform editor
It's already enabled and lots of people use it, although it's not compiled in on the mozilla-provided builds (but downstream builders enable it).
padenot··on AudioMass – free, open source, web-based Audio and Waveform editor
Hi, I work at Mozilla doing audio programming for Firefox, I see in your about page that you have issues with the Web Audio API, ping me (same nick a bit everywhere on the web, and also @mozilla.com) and we'll answer any question or fix any issue you have. Lots of things have happened in the audio space on the web since 2018!

In particular, low latency (like native roundtrip latencies, so <10ms easy, but depending on the OS) no-jitter/real-real-time audio programming is now something that developers can do. Lock-free/wait-free programming and SIMD are coming in the next weeks/months.

Very cool project in any case, I'll use it when I need to quickly do very high zoom on wave forms to debug things for Firefox.

padenot··on Coming to Chrome: a new way to use tabs
> Then you could add some sort of ctrl-p style fuzzy search that includes both url and content.

Try focusing the URL bar in Firefox and prefixing your query with '%'. https://support.mozilla.org/en-US/kb/address-bar-autocomplet... shows more prefixes like this. It does not search the content of the page, but is super useful still.

padenot··on Firefox 76
It's unlikely that AudioWorklet will fix a video call issue (zoom is alone in using it, because it doesn't use regular WebRTC facilities), but we're aware of performance issues, and they are being addressed.

In particular the graphics layer is being rewritten (the WebRender project), and exciting new developments are coming these days (in particular usage of the OS compositor to composite some bits of the page, that saves lots of watts).

This is particularly important for video calls, because often you're playing back multiple videos at once, so any inefficiency when compositing video is made worse.

Other things that are coming is to implement new ways to signal temporal or spatial scalability (telling other peers that videos flowing should be lower/higher fps, lower/higher resolution), this will also help.

padenot··on Firefox 76
Jitsi doesn't use AudioWorklet as of now (and doesn't need to).

We're continuing the work so that jitsi works well in Firefox, there are a few things remaining, but less so than a few weeks back (I'm more on the audio side, and those issues are mostly signaling or networking related).

padenot··on Jitsi Meet features update
Yes this is what I meant when I said "by the websites themselves".
padenot··on Jitsi Meet features update
Firefox has supported stereo opus for a very long time (four years at least?). We know it works, it's used by medical professionals for their job and they wrote a message a few month thanking us for this feature, that doesn't seem to work on other browser (according to them, but I see tickets open on chromium).

Of course all the chain has to be stereo, that goes without saying: input signal is stereo, negotiation has been done in stereo, having enough bandwidth is important (otherwise opus goes mono), and then playback has to be on stereo hardware (but that's the easy part).

padenot··on Jitsi Meet features update
This audio processing is trivially deactivable by the websites themselves. Instead of doing:

> navigator.mediaDevices.getUserMedia({audio: true}).then(...)

to get a stream of the input device, one can do:

> navigator.mediaDevices.getUserMedia({audio: { autoGainControl: false, noiseSuppression: false, echoCancellation: false }}).then(...)

similarly, an _existing_ input audio stream can have its settings changed while it's running like so:

> stream.applyConstraints({ autoGainControl: true, noiseSuppression: true, echoCancellation: true })

this for examples re-enables the processing that we put on voice by default.

This probably works everywhere, we've written a blog post about this that explain a few more things: https://blog.mozilla.org/webrtc/fiddle-of-the-week-audio-con....

If the website doesn't want to offer a control to switch this on/off, I'm confident this can be done by a browser extension in no time (which would have the benefit to work for all websites).

padenot··on Show HN: Zoom Redirector
This is incorrect. Zoom doesn't use webrtc audio and video channels, only data channels, and has reimplemeted audio and video channels on top of that, doing the decoding in js and probably wasm.

They also limit the resolution to 480p for the web app, probably because of performances. Browsers and zoom both use h264, but browsers usually use vp8 instead.

There is no reason webrtc cannot offer the same quality (or better) than zoom, at the same bit rate, but it all comes down to the actual implementation of browsers and web apps. webrtc-based apps work well, these days.

padenot··on Firefox 72.0
We're currently rewriting most of the graphics rendering of Firefox (with what is called the WebRender project). This is the first time Linux users will have hardware accelerated rendering by default, and is a pre-requisite to take real advantage of hardware-accelerated video decoding. It's enabled on some configuration in Nightly builds [0], and it's being worked on actively. If you want to know more, the graphics teams has a blog where updates are periodically posted about what is happening, often with interesting insight about the innards of a web browser rendering stack [1].

Once rendering is hardware accelerated, it makes sense to look into hardware video decoding. Doing hardware video decoding without hardware rendering roughly means we need to read back from GPU memory into main memory, composite the image in software, and then upload back to the GPU to display, which is super super inefficient.

[0]: https://wiki.mozilla.org/Platform/GFX/WebRender_Where [1]: https://mozillagfx.wordpress.com/

padenot··on Firefox 72.0
Would you mind filing a bug at [0] with the content of about:support and maybe a link or two where it doesn't work? Happy to have an initial look into this (I work on the media team at Mozilla).

[0]: https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&comp..., you can log in with a github account or create one

padenot··on Firefox 72.0
You'll be happy to hear that we've recently dramatically improved the audio quality of sped up (and slowed down) videos, here is the ticket for the details: https://bugzilla.mozilla.org/show_bug.cgi?id=1427267. This will be release in Firefox 73, in about 4 weeks (give or take a few days maybe).

HDR is very much in the list of things we want to do (especially for videos), but as you can imagine, it's not particularly trivial, and will require platform specific work, and deep changes into Firefox. On the other hand, hardware is shipping (we got some devices so that developers can start assessing the situation), and HDR content is becoming increasingly available, so we're trying to figure out what it will take to do it properly.

padenot··on Firefox browser will block the IAB's DigiTrust universal ID
Firefox already supports something like this if you go to about:config and flip `privacy.resistFingerprinting` to true.

It's not finished/perfect, but changes a lot of things.

padenot··on Firefox’s Fight for the Future of the Web
This is not at all the code to embed SpiderMonkey in Firefox, those are the bindings from DOM APIs to the implementation of those APIs in Firefox. Above that, calling into this layer, is another layer that transforms JS calls into C++ calls (roughly).

The interface between SpiderMonkey and the rest of Firefox is here: https://searchfox.org/mozilla-central/source/js/src/jsapi.h.

padenot··on Firefox 70
We don't do hardware decoding on Linux for now, because we don't do hardware acceleration for graphics by default on Linux.

Web Render might allow doing it properly, since the rendering is not on the CPU anymore.

padenot··on Firefox 70
Safari does not support vp8 or vp9 when playing youtube, and youtube serves h264 instead. h264 is less efficient in terms of compression ratio (more to download for the same quality), but h264 is decoded in hardware on OSX, and VP8 or VP9 isn't, which explains what you see.

This is why, for example, Safari does not have 4k video on Youtube, while being perfectly capable of playing 4k videos in general.

Depending on the machine, VP9 can be decoded in hardware on Firefox on Windows, but chip support is limited.

All that said, we're working on our video playback performance as we speak, especially on OSX (because it was so bad a few release back), but also in general.

padenot··on Firefox 70
Yes, Nightly is (I'm simplifying a bit but it's generally true) release + a few weeks of patches + different default settings, sometimes experimental things enabled, etc.

I'm running Nightly on OSX and I confirm all those improvements are there, but more are coming.

← PreviousPage 2 of 6Next →