jgraham

> Mercurial has the same "DAG of revisions" model as git, and works just fine, because it has a good UI.

People keep saying this, and this keeps not matching my experience of helping people with VCS issues at Mozilla.

Mercurial's model is roughly "append only list of commits, that separately form a DAG". This list model is leaked all over the place. For example the default `hg log` view just dumps that list rather than telling you anything about the history of the current working copy. To get that you need to add -f, which of course in most contexts means "do something unsafe". Worse, the list model is exposed in the local revision ids, which are prominently presented in the UI, and users have to learn that these ids aren't sharable with other users when discussing a revision. Also the most prominent "symbolic ref" in mercurial is "tip" which means "commit at the last position in the internal list", and so is sometimes the current head, and other times is some random commit from the remote.

Giving people help in mercurial is also super-challenging because, unless you are in some controlled environment, the first question is always "which extensions do you have enabled" and for complex tasks the next step is often "you should enable and learn to use <some extension>".

Which isn't to say that Mercurial is overall particuarly bad. There are some areas in which it's really an improvement over git (revset specifiers are pretty awesome, for example). But I often feel like praise for the UI comes from people who have internalised its flaws, and is repeated by people who haven't actually used it, but dislike git and are happy to accept something that confirms their preconceptions.

I really don't think that mercurial is very good at hiding the internals. Consider a common operation; asking for the commit history. If you type `hg log` you get something very close to a prettified view of the internal datastructure by default; a list of all commits in the repository, in the order they were added, each annotated with a sequence id that's an index into the internal list. So a new user has to learn:

* hg log by default doesn't tell them anything about the current code they're working on; it's just a dump of everything. To get the history of the current working copy they need to pass -f which they are learning in other contexts means "do something unsafe".

* hg gives commits sequence ids that can locally be used in place of the commit sha, but must never be shared across repositories.

In practice at Mozilla I have seen people run into both these issues, causing much unnecessary confusion.

Pretty sure it must be percentage of the 1997 value, otherwise it's hard to understand how the scale can accomodate both house prices and population in a single set of units.

People might also be interested in the Guardian article [1], which concentrated on the precipitous decline of Puffin numbers in Shetland. Compared to places where the drop has not been so steep, it seems that the birds are having to travel much further to find food (hundreds of kilometers in some cases), which is linked to changes in sea temperatures as a result of global warming; as sea temperatures change plankton are not available at the right time to feed sand eel larvae and so there aren't so many sand eels for the puffins to eat.

[1] https://www.theguardian.com/environment/2018/jun/03/shetland...

So rather than just blaming authors for their choices, it seems more worthwhile to look at the structural issues that lead people to make certain choices.

The Free Software movement was originally aimed mostly at personal freedom; the freedom to take something you owned and modify it. However in the intervening period it hass become integral to commercial tech. In a commoditise-your-compliments like strategy ("commoditise your dependencies", perhaps), there are now trillions of dollars of businesses that depend on open source software for their operation, but typically extract revenue from something proprietery built on top of that tech (e.g. surveillance advertising). And these companies are eager to release yet more open source software for critical (but not revenue-generating) parts of their stack. This has two key advantages for the company: it enables some of the cost of maintainance to be offloaded on the community, and anchors the cost of that component at zero. Take TensoeFlow, for example. If that's closed source then Google have to do all the work of keeping up with the state of the art, and can suffer if someone creates a superior alternative that can't be integrated into Google products. By opening it up they make it much more likely that new developments will be integrated into TensorFlow rather than into a competitor, and make it that much harder for a dirruptive competitor to usurp their position, because anyone looking to fund the development of such a competitor likely needs some revenue stream other than selling their software to finance the work, since they have to compete with free.

Now look at this from the point of view of someone developing some software. Unless you are targetting some niche market it's hard to make money through your software; by the mechanism above the expected price of such software is anchored at zero. Also many developers are aware that they have benefited from Free Software / Open Source and are keen to give something back. So creating software under an open source license makes sense; if you want your project to succeed the most valuable thing is to attract users who provide both motivation and contributions. In this way choosing a very permissive license is natural. And ususally this isn't a problem. But it does occasionally happen that projects are hugely successful, people want to continue working on them, and the funding available isn't commesurate to the amount of work that entails. In this case projects can be left struggling looking for a business model and a licensing decision that made perfect sense when the project was small can — from the point of view of securing funding — look suboptimal.

None of which is to say that Open Source (or Free Software) is a bad thing; I certainly don't believe that. But it does distort the market for software in an interesting way.

> Just another BDFL, that's it.

I think a lesson to take from this is that the concept of a "BDFL" doesn't really stand up. Putting an individual in charge dramatically increases the chances that they will suffer some kind of burn out compared to a situation where there are more people to share the burden with. That means the "For Life" part is more like "until they decide it's not worth the grief".

It's also clear that the "Dictator" part isn't really true; the stated reason for Guido leaving the project was the difficulty he had in getting the wider community to agree to a change he was in favour of.

The idea that having a single individual in charge produces a consistency of design seems false over the long term. Each time you burn through one leader you have to pick a new one who might have a relatively different aesthetic sense or set of priorities. And individuals change. Someone who was indifferent to functional programming, for example, in 1995 could be a huge advocate of it 20 years later, making quite different design decisions as a result.

Instead I would strongly encourage the Python community to look at what the Rust community have been doing. Leadership there is group-based, and the core team sets out an explicit vision for the project each year. This allows feedback from the community on direction and enables changes to be assessed against the explicit shared vision (vs assuming that vision exists in someone's head in the BDFL model). Ther exists a process for making changes that ensures that all (unique) feedback is considered before making a change, so that when decisions are made they're made based on shared facts. There is also emphasis on growing contributers into leadership positions, which togther with the team-based approach provides continuity when people move on from the project.

I'm not claiming it's a perfect system, but it seems to be working relatively well in a language that is going through a much more active phase (in terms of adding features) than Python.

It is true that very occasionally features are removed from the web platform. It is also true that the incentive structures that exist hugely favour not removing features or otherwise breaking existing content. For a feature to be removed there needs to either be an incredibly compelling reason (e.g. a security problem), preferably so obvious and widely agreed on that vendors are able to make simultaneous releases containing the change in order to mitigate the effect on the competitive landscape, or there needs to be a negligible measured usage of the feature. The latter is most usually the case when the feature was never widely implemented in the first place e.g. some (but not all) vendor extensions.

If it were easy to remove unpleasant features from the platform, browser vendors would be the first people clamouring to make it happen. For example, off the top of my head, there is widespread agreement that it would be good to get rid of, or otherwise improve in backwards-incompatible ways:

* Sync XHR * document.open * document.write * AppCache * localStorage

But that can't happen, because the desire of browser implementors to simplify their implementation is less important than the need to keep sites running. Apart from anything else, as stated uptrehad, breaking sites is a sure and certain way to get people to switch to a different browser.

Talking about "the issue" is a misunderstanding of how performance of complex software works. Overall performance is the result of the entire codebase, and there can be a varieety of ways that performance problems can manifest for individual users. One of the most productive approaches to improving performance with Quantum was fixing lots and lots of small and medium sized issues across the product that individually didn't have much impact but overall added up to a significant improvement.

There can be all sorts of reasons that specific sites or specific installations might show performance problems that aren't representative of the typical experience. Your specific environment might have some feature (particular hardware, profile data, addons, etc.) that happen to trigger a bad case in some code and so lead to an overall slowdown. Or you might be regularly using a site that happens to do something that's CPU intensive in Firefox but less so in other browsers. This could be a Firefox problem (maybe we implemented a feature using an algorithm with different tradeoffs compared to other browsers and it so happens that everyone avoids the worst case behaviour in Chrome, but not in Firefox, maybe our implementation just has issues that can be fixed), or it could be a site problem (sometimes sites just send buggy, broken code to specific browsers that causes them to use lots of CPU for no reason).

One of the things that was most valuable during the Quantum project was the work to improve the profiling tools. If you can capture a profile using the super-easy-to-use gecko profiler[1] and create a bug with the profile attached it should be possible to figure out what exactly is causing the problem in your case. Without that data it's really hard to make progress because the kind of problem you experience is just not something that would be allowed to ship if it was known.

[1] https://developer.mozilla.org/en-US/docs/Mozilla/Performance...

It seems to me to be overly simplisitc to take a specific event and say "that's the decisive moment where it all went wrong".

Technically Presto had some unique features; for example the inturruptable script engine allowed the browser to feel performant and responsive without having to heavily invest in parallelism via multiple threads or processes. But it also had some architectural differences to other browsers, and never had the market clout to ensure that the Opera-unique features were reflected in platform features and so had to be implemented by the competition, or even to ensure that features that were hard/impossible to implement in Presto did not become required for web-compatibility. For example Presto was unable to implement beforeUnload without a significant rewite of the core document loading pipeline, but that omission from Presto wasn't enough to prevent sites depending on it when it worked in Gecko/WebKit/Trident. Similarly a lot of effort would have ben required to port Presto to multiple processess (similar to the multi-year "e10s" effort for Gecko).

Presto was also highly optimised for memory consumption and so was ideal for running in resource constraimed environments like early smartphones and games consoles. But I think the launch of the iPhone and Mobile Safari changed consumer expectations for the web experience on mobile and Opera didn't manage to respond in an effective way. I have no idea what the optimal response would have been, but if Presto could have achieved double-digit marketshare on high-end mobiles (as opposed to low-end devices running Mini), we might have avoided many of the compat issues that currently affect the mobile web.

Organisationally I think there were other issues; I already spoke about the focus on the particular use case of making a highly integrated, highly configurable, desktop browser product, which doesn't look much like the more successful mass-market browsers today. I think later there were other problems, but I was far away from the executive decision making, so maybe I'm not best placed to comment on what the actual company goal was.

(I worked at Opera during the WebKit/Chromium transition and work at Mozilla now. But in both cases I'm just a normal individal-contributor type employee with no special insight into strategy or decision making).

There are always options. Opera could have doubled-down on Presto; putting more people on the core team, and focussing efforts to keep up with, and surpass, WebKit/Gecko. After all, that's basically the option that Mozilla took, which has resulted in Firefox Quantum. Would that have worked? It's hard to say. I would guess not, but I'm not sure the alternative really did either. Maybe Opera was already too far down the web-compat death spiral to engineer a way out. I don't know of a long-term bet like Rust that could have come good at the right time.

Certainly the top-level culture of the organisations was different; Opera's leadership were very concerned with maintaining/maximising the value of their shares, whereas Mozilla is more clearly driven by ideological goals around the success of the open web. Opera also had a (historically well justified) belief that they could do more with fewer engineers than other compaines. That seemed to work up to a point, but once the difference in resources became too great it was hard to change the approach.

Certainly one lesson is that it's hard, maybe impossible, to be a niche browser with a unique rendering engine. That is arguably a failing of the web, but nevertheless it's a strong indication that arguments that e.g. Mozilla should aim Firefox at small ideologically-driven markets are dangerous. One interpretation of the Opera history is that they were too focussed for too long on the subset of users who wanted a browser with lots of features and configuration possibilities. A product that suits those people might be actively offputting to other users, so inhibiting marketshare growth when faced with competition targetting simplicity and sane defaults.

There has been a lot of progress recently on the web-platform-tests project [1]. This is a cross-vendor effort to improve interoperability on the web platform through testing.

Historically the process for building web-platform features has been to write a spec and then assume that implementations of the spec would reach compatibility in an ad-hoc manner by patching until sites worked. The big innovation of the last decade — still controversial to some — was the idea that it's OK to adjust the specs themselves when implementations have converged on some other behaviour or when the spec is otherwise wrong. The goal for the future is to use the same engineering discipline that you would use for developing software to developing the platform itself. In particular the objective is:

* Every change to a spec that influences browser behaviour must be accompanied by a corresponding test case. * Every change to a browser that affects a cross-platform feature that isn't already adequetely tested must be accompanied by cross-browser test cases. * The results of those tests must be visible to browser vendors in a way that makes it easy to identify the higest value bugfixes (e.g. cases where N-1 browsers agree and one is different, or cases where a spec change makes previously correct behaviour — which may not have shipped yet, but just be enabled on nightly builds or behing a flag — wrong).

Apart from the work of actually writing the test, achieving those goals involves a lot of infrastrucutre work to ensure that the cross-browser tests are well integrated into the development cycle of each browser and are able to cover as many scenarios as possible (testing often involves manipulating the browser in a way that is not exposed to normal web content). Despite the large scope of the project, I think it's agreed that modernising the way the platform is developed, and prioritising interoperability at all stages of spec and browser development is essential to avoiding existential threats to the open web in the long term. Certainly Mozilla put a lot of resources into post-hoc fixes for site-compatibility issues, and whilst there will probably always be some bugs that slip through, it will be much more efficient to catch those problems up-front before they ship to end users.

I sadly don't have data to backc this up, but I'm pretty sure we're already seeing the effects of this effort, with recent, complex, features shipping with fewer cross-browser issues that we would have predicted five years ago.

[1] http://web-platform-tests.org/

I'm pretty sure that SqlAlchemy has caused more grief and frustration than any other single library. After all if I didn't know about the excellence of SqlAlchemy, maybe I wouldn't get so cross when I have to do anything non-trivial with the Django ORM ;)

Joking aside, SwlAlchemy is very impressive software, and zzzeek deserves all this praise and more. Every time that there's one of these anti-ORM articles I feel like SqlAlchemy pre-emtively addressed all the substantive criticisms in its flexible, well layered, design.

(And in the interests of fairness, Django's ORM is also very good at making simple things simple).

FWIW my last project was in Rust, and the current one is in Python (with which I have more experience), mostly due to Python having some useful libraries for what is mostly an exercise in gluing together different external systems.

Despite the fact that I chose to do this in Python, I'm not at all sure I wouldn't have had a working solution faster in a language that had a good type system and a error handling strategy that made errors explicit and forced them to be handled. I'm almost certain that writing in Rust would have ended up with something that was more reliable in the long term.

(FTR I imagine that Go may also have been a reasonable choice for this kind of work, but I haven't used it).

That's mostly a horrible accident though.

In retrospect what we should have done is released version 1.0 about 7 years ago when the parser was more or less fully compliant with the spec at the time and just been prepared to bump the version for backwards-incompatible changes (note that at the time the conventions of semver were not as widely adopted as they are now).

What actually happened was we planned a 1.0 release, and so made a series of "final" pre-1.0 version numbers like 0.90, 0.99,etc. and then kept finding one more thing to do, coupled with a decreasing amount of time working on it, and finally spec changes that we didn't keep up with. There certainly wasn't any TeX-like intent to converge on a number, just a series of decisions that seemed reasonable at the time, but had an overall effect that was ridiculous.

Presto (the old Opera engine) had a mode called "delayed script execution" which would speculate even harder than the gecko/servo approach. As I understand it — which might well be incorrectly — it would continue treebuilding after a script tag, and then if the script ran a document.write it would check if it produced something that wouldn't affect subsequent tokenisation (or, presumably alter the treebuilding state) and if so patch the tree in-place rather than throwing it away and restarting.

It was never debugged enough to be turned on on desktop, but I think it was considered useful enough to ship on mobile in the days when mobile performance was significantly worse than it is today. In particular, on a network where request latency is high having to pause treebuilding for scripts to download can significantly increase the time until it's possible to paint anything at all on the screen.

Unfortunately the reality is that a double digit percentage of all sites rely on document.write [1]. Whilst it's true that a lot of usage is advertising, it's also used for functionality in sites like Google Docs. Perhaps with a lot of work you could get the usage down to single-digit percentage figures, but disabling it entirely is going to break an awful lot of sites that are unmaintained. For something like NPAPI, where there is a real ongoing cost of support, breaking old content — with a multi year timeline for migration — might be worthwhile, but I think it's hard to make the case that document.write is in the same category. Yes, it's an old and terrible API that causes layering violations, makes writing scripted parsers much harder, and ideally authors would avoid it. But making it go away would be a huge project with relatively low payoff compared to other possible uses of the same effort.

[1] https://discuss.httparchive.org/t/how-and-where-is-document-...

There seems to be a certain amount of confusion on this thread, so just to be sure that it's clear, Selenium itself will continue to work with future releases of Firefox. This means that existing WebDriver-based browser automation remains compatible with with Firefox 55+. SauceLabs', and other testing-infrastructure-as-a-service providers' support for Firefox will be unchanged. The only thing that will stop working is the Selenium IDE which is a XUL extension that allows writing selenium tests without writing any code. This is undoubtedly useful to some people, but as the original post says, there are alternatives in development that are targetting a similar niche, so the situation is not as dire as the title implies (I imagine the title is worded this way to reduce the number of duplicate bug reports that the Selenium owners have to wade through on the subject of Firefox 55+ compat. for IDE).

To expand further on Mozilla's ongoing commitment to WebDriver, we employ one of the editors of the WebDriver specification [1], are making significant contributions to the WebDriver testsuite, and are actively working on our geckodriver implementation to ensure that we have a fully-featured standards-complaint implementation as soon as possible. I also know that other vendors are working on improving the spec compatibility of their implementations, so I think the future of WebDriver is very promising, with fewer differences between drivers that aren't the result of fundamental differences in the browsers under test.

[1] https://w3c.github.io/webdriver/webdriver-spec.html

FWIW Mac jobs are also often capacity-limited on Mozilla infrastructure. It's not unusual to have thousands of jobs waiting on hundreds of machines, which leads to multi-hour queues at peak times.

Fundamentally the problem with CI testing on Mac is that Apple don't provide the components you want to build a large pool of machines at reasonable cost, so everyone ends up doing slightly mad things involving hundreds of mac minis (obviously the Windows situation also isn't great; this kind of infrastructure loves free software on commodity hardware).

Well, yes, a low level protocol is going to have at least the same features as a higher level protocol implemented in terms of it. However I think you should mention the significant disadvantages of using a nonstandard API:

* Doesn't generalise to driving other browsers. Using a standardised API gives you the possibility to run your automation against any top tier browser. If you are testing a website this ought to be a top consideration.

* Uncertain compatibilty story. I assume that the devtools protocol can change at the whim of the Chrome team. Using something standardised means that your tests and client are more likely to continue working.

* More limited selection of clients. WebDriver in particular has a large number of production-grade clients that are actively maintained.

I'm not claiming that WebDriver is perfect; certainly if I designed it from scratch it would look very different. And yes, there is a period of churn as it moves toward being a standard. But I think the advantages of cross-browser support are a compelling reason for it to be the default tool for remote automation tasks that are within the scope of its featureset. By all means, if you have something that cannot be done with a standardized technology look at the proprietary solutions. But I don't think it should be your first pot of call.

Oh, apparently there's already a flag you can pass to the binary to enable headless mode, and geckodriver already supports setting those, so adding a separate option seems unnecessary. I thought it was just an environment variable.

Would people find it helpful if I add a `headless` capability to `moz:firefoxOptions` in geckodriver so that passing in `{capabilities: {alwaysMatch: {moz:firefoxOptions:{headless: true}}}}` when starting a session will start the browser in headless mode, if supported? This doesn't seem like something that's compelex enough to warrant a whole blogpost to explain how to get it working.

[I was the original author of much of html5lib]

Yes, http://speed.pypy.org/comparison/ has html5lib parsing (an old, static copy of) the HTML5 spec as a benchmark. It's about 3x faster than CPython, which is to say, still very slow.

I agree there's a good argument that writing new parser code in C is not a great idea, but I don't think it's possible — or at least easy — to get reasonable parsing speed in pure Python. My ideal solution here would be bindings to a Rust parser e.g. html5ever, although there is a tradeoff there in terms of ease of distribution.

The main missing feature in html5ever for these purposes is the ability to construct the final parse tree in the rust code (c.f. lxml), without which the performance bottlneck becomes constructing the Python objects to represent the document. Of course a streaming API like SAX can be even faster, but often isn't all that useful.

This is true and they ought to have retracted that. However if you apply the principle that you should boycott — at least least refuse to pay for — all publications that have ever published a misleading article on an important topic, then you will rapidly run out of options.

If you want to take a moral position it should be based on an overall assessment of whether the world will be improved by a particular news source ceasing publication. Given the rumored state of the industry it's easy to imagine that in a few years we could be in a "post news" society, where reputable sources have not managed to keep the lights on and it's only the populist publications that are still in business. If the long term choice is between "support a publisher that once made a bad mistake in a technology story" and "The Daily Mail is the only source of news", well to me that's not a moral dilemma.

I think this slightly understates the "business development" problems with developing a new OS, and a mobile OS in particular. It turns out that there are certain apps that, if they are not available on your platform, are immediate deal breakers for huge numbers of consumers (to the extent that they will ask for them by name in stores and return devices that won't run them). Often these apps lack web frontends (for non-technical reasons).

Obviously there are technical workarounds that you can attempt but consumers are wary of "knock off" products. So there is a vicious cycle in which people won't produce the content that you need to get users until you already have users. Possibly one can break this by supporting Android apps, but you would have to make the experience as good as an actual android device, which seems both demanding and limiting (because of cross-app interactions via e.g. intents).

Well a key property of Selenium is that it's cross-browser. Whilst it's nice to imagine that the web is in a position where running all your tests in some version of Blink means that your site will work on Firefox/Edge/Safari, I fear that the reality is still rather different.

To understand the constrains we have here, a ballpark figure is that we run a million tests per commit across all platforms (this is based on counting test files, so doesn't account for the fact that some files can contain thousands of separate test functions, but also doesn't account for the fact that we don't run every test job on every push). On average we see around 10 test failures per push due to test instability. So "on average" tests fail about one time in one hundred thousand. Of course, in reality there are some tests that fail much more often than this (one time in a hundred, say) and many that ~never fail.

Whilst test raciness is perhaps the most common problem, we also see intermittent failures due to race conditions in the browser code itself as well as through infrastructure instability. However it's unclear that moving to Rust will make much difference anyhow; Servo still sees intermittent tests so merely eliminating data races in safe code is insufficient to fix this problem.

I would be astonished if Servo is usable as a mass-market web browser in 12 months. Writing a rendering engine is hard because there is quarter of a century of legacy to deal with. By the metric of "can I use this for 100% of my browsing needs" Servo is far from being complete.

Much of Mozilla's "quantum" work is actually about improving C++ code e.g. the Quantum DOM project [1], which is a drive to improve scheduling so that a single background tab, even in the same process, can't jank the tab you are currently using [2]. Importing servo components wholesale simply wouldn't help here because Servo doesn't have a sufficiently complete DOM implementation, and I'm not sure the implementation it has contains all the desired properties.

[1] https://billmccloskey.wordpress.com/2016/10/27/mozillas-quan... [2] I am occasionally taken to refer to this project as "Presto: the good parts"

Unfortunately the new title seems to be misleading in a different way. There is no "project to replace gecko with servo". There is a project to advance gecko so it is state of the art in browser engines. Much of the current state of the art has been developed in Servo, and the natural way to bring that technology to gecko is to integrate the components from servo directly. This has other nice properties like increasing the Rust-to-C++ ratio, which we believe will be a win for safety and maintainability. But it is nevertheless components, rather than the whole shebang. Servo, for example, only just got document.write support and still (as far as I know) doesn't have a correct representation of document history. So it is some way from being a production grade engine for use on the general web.

But they aren't landing patches on "stable", but on the upstream development branch (i.e. mozilla-inbound/central). Since it seems more reasonable to view tor browser as a release branch of firefox than as a development branch, it seems like they are using "uplift" in the opposite sense to the normal Mozilla usage i.e. they are taking a patch from a product branch and moving it to a development branch. In that context "upstream" might indeed be a less confusing choice of words.

On the other hand I think it's clear what they actually mean here, so probably not worth worry about too much.

That is very unclear. Some studies suggest it isn't, but a recent metastudy reached very different conclusions [1]:

> Bicycle helmet use was associated with reduced odds of head injury, serious head injury, facial injury and fatal head injury. The reduction was greater for serious or fatal head injury. Neck injury was rare and not associated with helmet use.

[1] http://ije.oxfordjournals.org/content/early/2016/09/06/ije.d...