Comment by andrewmcwatters | Hacker News Reader

Comment by andrewmcwatters | Hacker News Reader

Parent Full thread

andrewmcwatters

Yes, this is the closest thing we have today, but it is built in such a way that implies you have a fully functioning browser already, which sort of defeats one aspect of the need for such a suite.

They’re also tests that don’t actually point anything out.

Take the CSS2 tests for example, https://wpt.live/css/CSS2/, I can’t do basically anything with these. They don’t tell me whether or not a particular part of the implementation has failed. They’re all basically arbitrary tests that sort of cover the appropriate properties, sometimes, maybe.

The biggest glaring issue is that the test runners run in the browser. Well, if my user agent doesn’t have JavaScript… I can’t “run” the tests.

Further, there are some parts of the standard that can’t be automated simply because it’s up to user agents to decide how the final rasterization turns out. You can’t just compare rasterization output or even reliably look for element heights for verification based on the differences between font rasterization.

This is also why a part of the suite requires manual verification.

What I really need as an implementor is the ability to point my software at some .html files, get some failures, add more implementation, get less failures, and repeat. With, of course, the caveat being that you can only really do this for what is well-defined in the standards.

But you can’t do that easily with the wpts.

Hi, I work on web-platform-tests.

I'm not really sure I follow your complaint, so I wonder if there's a point of misunderstanding. The tests written for DOM APIs certainly require running javascript to execute the test — that's a given — but most of the rendering tests are reftests. These consist of two files, one using the feature under test and one avoiding that feature [1]. The test passes when the two renderings are identical. That does make these tests difficult to run "by hand", but in terms of implementation it's possible to automate in anything that's capable of rendering to an image. Reftests are preferred over comparing rendering to a fixed image because fixed images tend to be invalidated due to unrelated/permitted changes in the rendering (e.g. a different font or different antialiasing choices). These will typically apply to both test and reference and so are handled automatically.

Reftests obviously aren't suitable to bootstrap getting very basic things working, so there are some manual tests; I suppose one could hope to avoid that by inventing a special kind of output inspection automation just for those basic tests, but it's hard to justify.

IN general, however, the "the ability to point my software at some .html files, get some failures, add more implementation, get less failures, and repeat" is exactly what wpt offers. There is some work required to stand up the necessary infrastructure to run the tests automatically, but it's common for new implementations of the platform to make use of web-platform-tests (e.g. Servo and Flow have both made use of wpt).

In terms of coverage it's very difficult to demonstrate that all the requirements of a spec, and all its interactions with other specs, are completely covered. There have been various proposals for adding test metadata to try to measure coverage, but these have largely proved impractical. Nevertheless some CSS specs have links from the spec to the relevant test cases. And if you find cases that aren't covered by exisitng tests it would be great to get a PR to add new tests [2]

[1] https://web-platform-tests.org/writing-tests/reftests.html [2] https://github.com/web-platform-tests/wpt

I’m not sure what you define as “getting very basic things working” but there’s no acceptable threshold for me. Are line boxes very basic? Is overflow compositing very basic? It’s irrelevant whether or not one is bootstrapping.

It is unacceptable to me to use the software I’m testing to test itself if I can’t establish a baseline for the rest of the tests.

Otherwise, all of the tests could be invalid one day based on a regression and I wouldn’t know. But this is explicitly done with reftests.

And I don’t know what you’re talking about with the DOM API tests, there are several layout and compositor tests that require JavaScript and it’s unclear why other than the test author decided to use JavaScript to verify DOM metrics. That’s fine, I just don’t want that in my layout tests.

I’m OK with a subset of the tests just not being automated based on font rasterization and line height calculations. The standards make those expectations clear, but everything else feels like fair game to me.

I can either test for it, or I can’t. If I can’t, it’s not automated. And if it’s automated, I need information about what is failing, and much of the reftests don’t explicitly point out what they’re testing. The assert data is useful when it’s available, but even then sometimes test authors neglect to point out explicitly what they’re testing for.

Plenty of the layout tests in particular are just box model renders that should match with no explanation as to what section or text is being tested against. That’s just not acceptable. It’s not even about interactions between specs, the specs isolated are not well tested.

If you’re testing, let’s say, the width calculation of a box based on children boxes’ widths in normal flow, it needs to say that. And I think I’ve only ever see some of WebKit’s test’s help links reference actual sections.

Otherwise the WPT provide little to no value to me.

My team has been better off writing our own tests because we directly refer to the recommended or living standards texts by section, quote it and link to it, and as a result there’s no ambiguity about what property, calculation, or raster output we’re testing for.

That seems unrelated to whether WPT could cover this bug?

This subthread isn’t discussing IndexedDB.

It isn't? It looks to me like this subthread started with a question of whether this regression could have been caught by automated tests: https://news.ycombinator.com/item?id=27511318

Reply on news.ycombinator.com