Not all automated testing tools support Shadow DOM in web components
matuzo.at
matuzo.at
I took an alternative conclusion - don't use shadow DOM or other bleeding edge tech when something much simpler will do. What ever happened to using document.getElementById? You can even skip much of this if you finish your work on the server before sending that text/html response...
I sometimes wonder if there's a huge thing I'm missing, but then I remember our customers love our product. Not a single one of them can tell that we don't use react or WASM. In fact, we have yet to receive a single complaint regarding our multipart form posts resulting in actual browser navigation. For whatever reason, the users don't seem to give a shit about this. It only ever seems to come up in the most ridiculous nerd conversations.
Declarative Shadow DOM, more recently added and not yet universally supported, fixes one of the two biggest problems with it (that you have to load and run scripts before it can work), but the other one (that it’s strong insulating encapsulation, which is just occasionally what you want but normally not) is by design and is not going to change.
In the forms they’ve ended up in, Shadow DOM was a mistake (yes, they should have just left browser widgets like <select> and <audio controls> special), and Web Components were almost certainly a mistake.
I’m a user. I want to write user scripts to tweak things to my liking and to support my specific use cases. If you use shadow roots (especially closed ones), you may fundamentally prevent me from doing so, removing the agency of my user agent. I don’t like that.
The tools you mention have helped enable a ton of new forms of development and interactions on the web.
Consider: I was recently baking a cake and didn't have to use a sous vide for anything, I don't understand why people keep hyping those up so much. That's similar to how your comment reads to me.
This analogy seems to work pretty well. On the other side of the fence, I see people talking about how good their sous vide cakes are and how they’d never work at a bakery that didn’t exclusively use a sous vide.
That’s not to say that the cooking method is never viable, just that it’s touted as the only thing necessary. Meanwhile online baking recipes list “make sure you own a sous vide” as the first step.
Are you sure you understood the analogy? The components of it were:
1. Person says they built a basic website without React, doesn't know why people use React (it's because the use case for React typically isn't "build a basic website given unlimited time and effort")
2. Person says they made a cake without sous vide, doesn't know why people use sous vide (it's because the use case for sous vide typically isn't "bake a cake")
Assuming you interpreted this from "online baking recipes", I was intending to refer to tutorials that explain something like "how to disable a button" and step 1 is "npm install $POPULAR_LIBRARY".
That's the point: tutorial sites largely don't do those things.
As for the separate discussion of whether they largely suggest React as a first option for building a given web app, there's no problem with that: it's entirely possible that the best path is to try React first, then look at other, lower-level options only if necessary.
The problem is only if they largely say that's the only option, which they don't, so what's the problem?
1000% agreed.
The part where this breaks down for me is the observation that many developers don't seem to have conducted the barest of due diligence regarding what tools and options are even available.
I can accept that you went with ReactJS, but not if you are operating in a vacuum where access to MDN is restricted and you didn't even consider the merits of a "no-build" web ecosystem.
Personally, I believe if they found something that works, more power to them. In all likelihood, the "no-build" option is likely more time and more work than using frameworks, for the sort of non-basic web apps someone might choose them for.
Doing a comprehensive due diligence of all the possible ways to build something is, imo, the sort of analysis paralysis that stops hackers from just going out and building. Yes, you could become an expert in the full stack, such that you could build an app omitting any or all of them. Or you can build a thing. You're going to throw the first one away anyways.
Just out of curiosity, how do you handle errors, specifically timeouts here? I'm guessing the HTTP server serves a custom error page, does it have a reference to the previous page with all the data they tried to submit and so on? Probably how I would have handled it.
Timeouts are a bit trickier, as the user would just see the browser error page and dassit.
Nice thing with decoupling server responses from what the browser actually renders is that you can handle things like that exactly like how you (as a developer) want to, without breaking the current page.
But, as always, it depends. Seems you folks are giving your users a lot of value from whatever you're doing regardless, so seems what you're doing is working :) Congrats.
Seriously, scripting-based submission, navigation, &c. is reimplementing a lot of what the browser give you for free, and it can’t be done perfectly (e.g. tab loading indicator) and what is reimplemented is absolutely never as flawless as possible. Yes, you get some interesting advantages, but you lose a lot too, more than developers tend to admit.
Many users have no idea what does that mean and will just panic seeing the default error page. They will not read the message there and either will abandon the form, reach customer support or will restart their flow. You are absolutely right about the complexity of the form submission when network problems are taken into account, but default browser behavior isn't the best solution.
Today most people do not encounter those pages, because they interact with SPAs and more likely to encounter different error behavior. Good internet connection won’t prevent anything. Imagine a child accidentally unplugging the wifi router and grandma seeing the timeout page, because her browser doesn’t know that wifi is down. What she would do? This is the kind of situation IT support is dealing with regularly.
We don't.
We are using stateless HTTP semantics throughout. There is no web socket or other streaming abstraction. You can key in a form, come back tomorrow and submit it and the server won't care (assuming business logic checks out).
If the system is down at submission time and someone wants to retry after it comes up? No problem. The browser will give you an opportunity to resubmit the form data since the prior POST timed out. This has been a thing since ~forever.
The end user experience when the service is completely down is that the browser displays "connection to server timed out" with various initial troubleshooting steps that the vendor manages. Either the server is up and everything works, or it is down and nothing does. Whatever partially-available thing we might be going for doesn't make much sense in my head.
Most of my differentiation is in the UX: specifically things like animations that were carefully selected to make the UI feel snappy even though the AI isn't that quick.
It sounds silly, but there are literally 100 apps doing exactly what mine does. If my site felt like submitting a form, and the UI had native selects, and overall it felt like a site that leaned more on native web technology, the demographic that uses it would have shunned it for other options before giving it a chance.
Sometimes you're in a market where you get the luxury of ignoring looks and just delivering value in a way that doesn't allow people to complain, but other times you have to give the people what they want: and a lot of them want that "it feels like an app" experience.
A little bit of javascript/AJAX/etc is perfectly acceptable. We didn't get 100% of the way there with multipart/form because iOS/Safari exists and will crash hard if you try to post big forms.
You don't need ReactJS to submit a user input and avoid a page load. This is ~20 lines of vanilla javascript.
I find things like https://stackoverflow.com/questions/10195459/ios-safari-cras... and https://bugs.webkit.org/show_bug.cgi?id=84168, from 2012. Is that still an issue? I find it difficult to imagine you could have a bug that prevents you from submitting a form with even half a megabyte of data being allowed to last for long.
Yes. It is 100% still an issue. That is the exact bug I was looking at. I can get Safari on my latest gen iPad to crash hard after only 3 submissions pretty much every time. We are sending large images in our forms but it's only like 1-5 megabytes.
(I guess the size threshold has increased as device memory has increased.)
This tells me that very few developers are doing things "the old" way or otherwise questioning mainstream technology narratives.
I sometimes find myself wondering if multipart/form is going to be deprecated soon. If you look at the latest Azure Function isolated worker stack, this hasn't even been dealt with yet. I had to grab a 3rd party library to parse these submissions out of my request bodies.
If you have a fancy web socket use case, no problem - we got your back. Want to build a boring LOB business app? Better be ready to start shimming those HTTP post mechanisms.
There'd be a lot of "just a little bit of vanilla Javascript and CSS" sprinkled all over that'd all need to have state managed in a manual way that wouldn't be a meaningful improvement over React.
Like I said, I'm working on something where seemingly minute UI/UX details matter to the target user. So I can't just say "CSS and Vanilla JS don't support that without pulling teeth, it's a small thing, so I'm going to ignore it". Sometimes that's going to be a valid strategy and sometimes it's not.
Passing element references is more compatible, but more verbose. And in the case of test framework POMs, a poor choice - because it leads to stale element references in a reactive or virtual DOM.
Some libs like Playwright are re-living the jQuery timeline and building enhanced selector engines that pierce shadow DOM by default.
We're forced into a future where your toolbox either contains shadow DOM or querySelector.
Shadow DOM's adoption problem comes down to ergonomics. We prefer jQuery's vision of the web, grabbing elements with handy one-liners. Shadow DOM broke that. And the practical benefits of shadow DOM have never outweighed the convenience of selectors.
Our ability to use a testing framework like Playwright without having to resort to a bunch of weird adapter code is a big part of why we went with dumb-simple web technology. Their recording feature "just works" for us and we get perfect replay 100% of the time.
It’s worth noting, for those that don’t realise it, that from client-side scripting you can’t reliably pierce the Shadow DOM. Playwright must be doing this with privileged APIs.
Specifically, although element.attachShadow({ mode: "open" }) sets element.shadowRoot to the ShadowRoot that it returns, element.attachShadow({ mode: "closed" }) only returns the root, and you can throw that away or store it in a private property or closure variable or whatever, so that it’s impossible to access from outside. Yes, shadow roots give you an inviolable membrane (and one that I don’t think content blocker are piercing yet, because not many sites have been abusing it yet). The only way outside code will even be able to discern that there is a closed shadow root is by trying to create one and having that fail (which is a canary, you can only do it once).
The vast majority of devs I've worked with who try and use them are basically using them like JS frameworks or HTML templates in normal apps.
Shadow DOM is just really a poor fit for this type of use IMHO and results in practical issues like the OP.
That is partly my point then - every time I've run into this with devs, the defaults, docs, comments, etc assume Shadow DOM is needed/important and then things like the OP bite them.
At the start, actions from good unit tests in the browser execute user simulated events against DOM nodes the user has access to. That is all, because that is the actual user experience If your testing approach can do this directly, as in without framework magic, it will work the same with your favorite framework and vanilla JS and it will execute much faster.
Then test assertion is as simple as validating changes to DOM nodes, or no changes in the case of negative tests. Such assertions can include the presence and/or visibility of a DOM node, value of text, the size and/or location of a DOM node, presentation, node order, and so forth. User facing events include absolutely everything a user could do with a mouse, touch, or keyboard with only exception to mouse cursor movement.
If that is no clear do nothing else with testing until the above is ironed out. Once the above is solidified more advanced testing can determine whether events/listeners are attached as expected or whether hidden data/metadata is available to the page as expected.
Also, the major common testing tools make use of Selenium styled remote access to the browser via means like CDP or W3C's WebDriver protocol to backdoor a browser. These are really complicated, way more complicated than you need if testing on a site you own. Testing is far more simple, and secure due to same-origin policy, if you conditionally embedded a test receiver script into the page and beacon out test assertion results via WebSocket.