Help the Graphics team track down an interesting WebRender bug
mozillagfx.wordpress.com
mozillagfx.wordpress.com
Based on the symptoms and difficulties to reproduce, I think what you see on the screenshots are incomplete renders. The FF code submitted some draw calls to GPU, and copied the render target texture without waiting for draw calls to complete. A good way to fix is use a D3D11_QUERY_EVENT query to wait for completion of rendering.
If all GPU access happens through D3D11 this shouldn’t happen i.e. the API guarantees to wait for the completion of previously submitted draw calls. It may happen in practice when mixing multiple GPU APIs, e.g. when using DXGI surface sharing to pass textures between D3D11 and DX9. Also, it may happen when using D3D from multiple threads. Easy way to detect the latter, set D3D11_CREATE_DEVICE_DEBUG flag when creating the device, and read warnings in debug output of the process.
Another possibility is just bugs in rendering code. To troubleshoot them, https://renderdoc.org/ is awesome. Unfortunately, if the FF doesn’t use D3D to present, only rendering to textures, some changes to FF’s source code is required to capture frames with RD, see this page for details: https://renderdoc.org/docs/in_application_api.html
Another post mentioned 2x4k screens so I thought creating load might delay expected events so I set up gpu to lowest:
"echo low > /sys/class/drm/card0/device/power_dpm_force_performance_level"
Then opened 2 drm and lot of youtube videos with new tabs and keep refreshing reddit, after a minute or two got black boxes
specs here: https://pastebin.com/JXE2HFUu
EDIT: Nope its not it turns out marvel has black theme, sorry misleading comment, move along
I have no idea how exactly FF uses GPU. However, all 3D APIs, especially modern ones, are just ways to access the functionality of underlying GPU hardware, which doesn't depend on the OS.
If FF uses Vulkan, the direct equivalent of that query is probably VkFence, but unlike D3D11 VK has other ways too, subpasses and VkSubpassDependency, and maybe something else I forgot.
"WebRender is actually using D3D through OpenGL on top of ANGLE"
maybe an angle bug, then.
There is work to port Webrender to gfx-rs which abstracts over Vulkan, Metal and modern DirectX but it seems to only live in the fork of the Hungarian Szeged university for now: https://github.com/szeged/webrender/issues/198
Maybe then Firefox can purely use DirectX?
- [1] https://github.com/szeged/webrender/issues/212
- [2] https://github.com/szeged/webrender/wiki/Optimization-Status[1] https://github.com/szeged/webrender/issues/212
[2] https://github.com/szeged/webrender/wiki/Optimization-Status
It would be nice to have a native D3D11 backend (and D3D12, and Metal, and Vulkan) someday, but that day isn't today. gfx-rs, or wgpu-rs, looks promising as an abstraction layer over all of these APIs.
It's opensource with MIT license. I wouldn't expect too many issues supporting such use case, given RD's support for normal OS-supplied GLES. I would only expect complications if you'll try capturing both layers at the same time, GLES between your app and angle, and D3D between angle and d3d11.dll.
Alternatively, could you emit markers at or above the Angle level to provide an easier to understand RenderDoc trace?
Bugs are usually quickly fixed once someone on the graphics team manages to reproduce them (in a renderdoc capture). Unfortunately nobody has managed to reproduce it yet even though the issue has been in the team's radar for a while.
I __only__ experienced the text tearing. It would basically be a slash through the screen, usually two or three where text would just be displaced. A small scroll would fix it.
This has happened to me in both Manjaro and Ubuntu and on two different computers. The Manjaro is a Inspiron with a 1060Qmax. The Ubuntu has a GTK Titan Black (so old).
I have not experienced these glitches in fairly recent memory. So probably last two versions of FF. These computers are my daily drivers and I keep them fairly up to date.
Usually where these glitches would happen is in long form text articles. I don't recall seeing them on Reddit, but more like NYT. Hope this helps.
It doesn't happen too often, but iirc I saw a text glitch sometime in the past few days. Moving my mouse over the glitched area or a small scroll solves it for me too.
I have a GTX 1070.
Did you think bugs got fixed by mentioning "me too" in a HN comment thread? :)
the good: it used to crash my pc when the render process would just hang at 100% after resume, doesnt anymore
the bad: sometimes i have to scroll the page and tab bar after resume now
progress
MBP 2013 with integrated graphics, running Mojave and the latest stable public release of Firefox. I haven't messed with about:config.
However gfx.webrender.enabled is set to false so are they even active?
FF 72.0.2
This was on windows 7 64bit with a probably outdated install because my favorite extension for vertical tab tree was always behind on version support.
This error would manifest when I had many tabs open for a long time. I'd often keep sessions of hundreds of tabs open for weeks. The errors would start to appear and gradually get worse until I had to force crash ffx and restart (and recover my tabs!)
Not really actionable or reoro, but anecdotally maybe it helps to hear this isn't strictly new.
It could, actually, especially if it's not well maintained.
I've never seen this in Firefox but makes me wonder if its windows related instead of FF/Outlook specific
For what it's worth, I downgraded my drivers to the last set of drivers from 2019 (19.9.2) and the problem seems to have gone away, but I'm having other issues instead (for instance, half the screen flashes dark when Night Light turns on). Guess it's time to upgrade and file some bugs.
The state of the art in debugging GPU issues is to have a closet full of random hardware purchased used on eBay that hopefully contains something close enough to what you need to reproduce the issue you're working on.
Most fancy debugging tools pretend that GPUs don't exist and GPU-focused debuggers are incredibly buggy and feature-poor, not to mention hardware specific in many cases.
With open source drivers, AMD or Intel could just walk in and steal whatever optimizations nvidia has made for a game.
> The specification must include a written description of the invention and of the manner and process of making and using it, and is required to be in such full, clear, concise, and exact terms as to enable any person skilled in the technological area to which the invention pertains, or with which it is most nearly connected, to make and use the same.
You don't have to provide code, but you have to teach the reader of your patent how to make and use the invention.
It does? Exactly zero of the companies I've worked for in my career have forbidden me from reading patents.
Given how many patents are vaguely worded, and how, um, interesting the decisions in patent cases sometimes come out, many companies decide that it isn't worth the risk.
Note well: IANAL. I've just been in companies that had this practice (don't read patents), and I remember why.
The USPTO is terrible at performing prior art searches and terrible at rejecting obvious claims, so there are legions of trolls waiting in the wings to jump on them if they go open source.
This is not true. Driver quality is a major differentiating factor between GPUs and absolutely influences purchasing decisions.
GPU makers don't want their competitors to be able to take parts of their drivers and benefit from them. They want to protect trade secrets. They want to avoid revealing how many of their competitors' patents they are violating. They don't own the rights to some third party code or trade secrets that are incorporated in their drivers. They want to avoid revealing future product plans.
Of course it's debatable whether closed source drivers actually help with all of the above reasons, or whether they are good reasons to begin with. But those are the reasons.
Opening the source to drivers as they exist now might expose IP that might not be yours to share, IP that you aren't licensed to use or open you up to liability because of how bad your drivers actually are.
That said, I do believe hardware companies should prioritize open-source drivers to benefit themselves and their customers.
It may on may not end up improving support, but you can't really know that ahead of time. Then you have the risk of people using your code against you, finding vulnerabilities, patent/IP violations or simply reverse-engineering your hardware more easily. You also won't be able to cheat in benchmarks anymore.
There's a cultural aspect to it too. Paying all these engineers to write that code was expensive, and now you want to give it all away for free? Most companies never do that for anything, why would they make an exception?
It's dumb, I too would love for these drivers to become open source, but it doesn't look like it will be happening any time soon.
I'm not saying it doesn't exist, but I've been doing professional software development for more than twenty years and have never encountered this. There's plenty of "we should be paid for our work and innovation" but nothing so reflexive as what you describe here.
Your second point is correct, that open source drivers do get better support. AMD has better support on Linux than NVIDIA because it has an officially supported open source driver (AMDGPU). However, the support they receive from open-sourcing the drivers is not always the R&D required to gain a competitive performance advantage, limiting the competitiveness of an open source driver (at least compared to proprietary).
Would there be some sort of uproar if people found out that Firefox had special optimizations for common websites, for instance?
> Would there be some sort of uproar if people found out that Firefox had special optimizations for common websites, for instance?
It would ruffle a few feathers if Chrome was optimized for Google properties, but ran their competitors' code with a performance penalty.
One of the issues in United States v Microsoft was that IE used optimized APIs that Microsoft's competitors, like Netscape, didn't have access to.
It is: https://9to5google.com/2018/07/25/youtube-chrome-firefox-edg...
As long as they aren't actively sabotaging your software, I don't see why this is a problem. Adobe is clearly the market leader in photo editing. If adobe pays nvidia to improve performance of photoshop, why is that an issue? They aren't paying nvidia to make gimp worse.
Also "sweetheart deal" is a loaded term. What is your definition of sweetheart deal here?
> A "sweetheart settlement" may also occur in a legal context. For example, in a class-action lawsuit the attorneys representing a class of plaintiffs may reach an agreement with the defendant in which the primary result is a lucrative fee for the attorneys rather than maximum compensation for the class members.
> Noted instances and allegations
> In a 1947 unionization dispute, San Francisco area grocery store owners claimed that other stores who had "given-in" to union demands had signed sweetheart deals with the unions.
> In a 1949 dispute between the rival American Federation of Labor (AFL) and the Congress of Industrial Organizations (CIO) involving unionization of the laundry industry in Indianapolis, Indiana, a lawyer for the 42 laundry and dry-cleaning plants testified before the National Labor Relations Board that an AFL union contract was not a sweetheart deal between the employers and AFL union officials, as alleged by the CIO.
There was a comment on this site a few days ago that claimed Google was being anti-competitive by hard-coding a list of their web properties into Chrome for it to send diagnostic info when visiting them, so yes, anything where there's a special path for some 1st/3rd parties but not others could be construed in many ways that look bad for the author.
But embedded GPUs are so fucking obnoxious. There’s about a million of them and they pretty much all have proprietary binary blobs. Pretty worrying for something that could potentially allow any device to be remotely compromised via some malicious webGL or something. Good luck getting the manufacturer to continue supporting it 3 years from now though
No, Not at all.
If we go all the way back to 3Dfx era, we have seen dozens of GPU ( or 2D/3D Graphics Accelerator before the term GPU was coined ) trying to compete in the PC space, 3D Labs, Matrox, PowerVR, Intel I740, S3, ATI, Nvidia, Rendition, ( These are on top of my head, sorry if I missed any of your favourite ). None of them had much problem with designing hardware, their maximum triangles, ( If I remember correctly it was one of the unit used to market graphics card before GFLOPS were used ) were all on par with each other. And some even exceed the current dominant players, but in the end they all failed.
Why? Because they cant get the drivers / software to optimise the use of their Hardware. They do not perform any where close to their potential in Tomb Raider or DOOM. There are huge amount of optimisation built up over the years. As Nvidia once said ( That was even before their huge bet on CUDA ), 60% of their engineers are Software.
To point where the I am going to say, the Driver IS the GPU. You are just paying the hardware to use it.
* And now I feel very old just typing this up. :/
You'd think that, but understand that Nvidia's primary advantage is actually in the drivers. In the graphics industry it's known that AMD GPUs often perform better at various tasks as they become highly optimized. However Nvidia GPUs most often benchmark better. As it turns out Nvidia drivers have an enormous set of optimizations it can enable based on heuristics which end up being the key to making up most of the difference.
> contain complete compilers that munge your code with a focus on speed over correctness
If you’re a developer, the right tools are available. Set D3D11_CREATE_DEVICE_DEBUG when creating the device, also run debug builds of your HLSL shaders. You’ll get huge amount of error checking and diagnostics this way, including very detailed and specific messages about incorrect API use.
> GPU-focused debuggers are incredibly buggy and feature-poor
I indeed found and reported bugs in GPUs and their drivers (mostly Intel GPUs), but it only was a few times over my career. The majority of times they were bugs in my code, not in GPU drivers, and renderdoc.org helped a lot fixing them.
I generally agree GPU debugging is harder than CPU debugging, but I don’t see how open source driver gonna help. It’s an unfortunate combination of execution model (many thousands hardware threads running your code in parallel), external piece of hardware completely separate and asynchronous from the CPU, and very limited ways to interact or even output data (you can’t printf messages, nor allocate dynamic memory, from GPU code).
Funny you say that. I've seen many developers praising the open source stack under Linux, as you can step right into the driver and fix it yourself, instead of landing in symbols land.
Feral Interactive mentioned it there, for instance: https://www.gamingonlinux.com/articles/interview-with-feral-...
the Servo/WebRender folks maintain such a list of driver bugs with workarounds, you may find useful:
The majority of their Windows bugs aren't GPU driver bugs, they are bugs of compatibility layers like Angle and SPIRV-Cross.
IMO when using native stack i.e. D3D11 or 12 and HLSL, the quality of GPU drivers is usually OK on modern Windows.
It's 2020 and I still sometimes swap my renderer from OpenGL to D3D9 when I need to debug a graphics issue because the D3D9 graphics debugging tools are that much better than cutting-edge GL debuggers. (I'd use the D3D11 debugging tools but my 11 backend doesn't work right)
Pixel 3. Hopefully it's a common codepath.
Does the black go away by itself after a couple seconds, or when you scroll, or does it stay forever?
E.g. gpu has a frame pending, CPU updates glyph cache, GPU renders pending frame using incorrect cache data.
My fix was to turn off all the Dynamic Clock and GPU Boost stuff in the NVIDIA control panel.
Even nowadays, on rare occasion on Ubuntu 18.04, something like this occurs during startup of Android Studio. I don't recall seeing it on Firefox since at least 16.04 though.
It turned out a Windows updated over night and replaced my AMD driver with some old compatibility driver. After updating back to the most recent driver the issue went away.
It causes a race condition that results in proving incomplete data to the next step of the process.
There is surely logic to prevent it from happening. But the data initially used the bad routine so no one expects the race condition to occur.
And at the end we have a bad routine that processes data badly because data are bad and incomplete. This routine does not expect how it receives the data and in what time.
In a result rendering breaks, but for the logic of the code it seems fine. So the error block is displayed. Gor GPU rendering it is a black box.
What is interesting if you did deep enought to find all these 3 issues that cause it each of them separately is fine. It is what they do together causes the issue.
Very nasty thing to track. And also fix. Because on a high level there is little to about it. And a workaround is mostly to detect the situation and process data slightly different way to avoid the issue, or to just do processing two times in a rather hacky way.
I’ll try and take a screenshot next time it happens & report it.
Funnily, this alias is incomplete and therefore not enough to join. A room ID might have been enough, but just an alias without a server part is worthless, a bit like someone left an e-mail adress as gfx-wr-glitch@, without specifying the rest.
I've successfully joined trough #gfx-wr-glitch:mozilla.org, (roomid is !qaAtjMWizEgMcVSbNH:mozilla.org, no alias on matrix.org) and told them so.
Edit: they fixed it. I wonder how often that happens with those new to Matrix?
They run their own server, just like they ran their own IRC server before, and at first didn't really consider the need to make that explicit. I doubt there was any misunderstanding of the tech, or that the channel names are per-server.
I wish non-technical people understood this.
Or is this just an unofficial blog from people that is unrelated to Mozilla?
But yes, this is an "official" blog from Mozilla employees on the graphics team, of which I too am a member.
It surprised me that Mozilla is able to simultaneously have things like the MDN, basically the best resource for HTML and Web APIs, but then other references or publications scattered around everywhere.
Whenever I hear the 'mythical legend of the 10X rockstar developer', I think of them. Mozilla needs more of those developers to reproduce AND send patches to interesting bugs like this. Not the whining copypasta forum-posting 'users' calling themselves 'developers' who take many open-source projects like this for granted and don't appreciate the time spent by the maintainers of the project.
Nowadays they're usually doing this on a developer social network through every maintainers nightmare 'The Issues Tab'.
Regardless of community engagement, perceived interest, Github stars or whatever else — finding a developer willing to spend actual time and effort to make a project better is very, very rare.
For me it's not about the reputation boost - it's that I see a minor flaw and can offer a quick fix, rather than just moving on and pretending I didn't see it.
It's like picking up litter when you see it on the street.
That said, I'm not a maintainer of any open source project with actual users, so I'm not familiar with the kind of gamesmanship you undoubtedly see once a project is popular.
- Discuss their changes beforehand (usually on the issue page)
- Use patterns and practices that the rest of the codebase follows
- Listen to comments and feedback
- Write tests
While the work is "free", it should still be treated professionally.
Interesting. When I last looked at contributing to open source, some general advice was to offer up code even if its likely the wrong approach. The idea being that maintainers are busy, and there's lots of people that drag on their time without ever contributing anything. Showing you're actually willing to put in time and effort will make the maintainer more likely to guide you in the correct direction because you've already shown you're willing to spend time on the issue.
I see a somewhat similar bug on my Linux box, where windows flicker between black and normal, and the flickering stops if i focus on the window. I don't think it's specific to Firefox, though - i believe i've seen all sorts of apps do it. I do have an nvidia graphics card, FWIW.
If you want to reproduce it yourselves then perhaps try pointing a hairdryer from a distance at the various components until they start to create trouble, or alternatively just overclock them towards the breaking points.