Busiest hour ever in the history of Gov.uk 17M page requests just 21 5xx errors
twitter.com
twitter.com
Their websites might not look the part, but they excel (again, in my experience) in their practicality, accessibility and ease of use. And that is exactly what you would expect from a government website.
They had an analysis showing how many % of their visits had broken JS support and therefore the need to build for the lowest common denominator.
When a team builds a service using the service manual as a guide so many things become so much easier.
Need to explain something to a non technical manager? The service manual has got you covered:
> Progressive enhancement is a way of building websites and applications. It’s based on the idea that you should start by making your page work with just HTML, before adding anything else like Cascading Style Sheets (CSS) and JavaScript.
A great principle, and nice to see they even think about Lynx accessing the sites [0]
The post I was thinking about with the analysis of why JS/CSS shouldn't be assumed to work was [1].
[0] https://technology.blog.gov.uk/2016/09/19/why-we-use-progres...
[1] https://gds.blog.gov.uk/2013/10/21/how-many-people-are-missi...
Addendum: plus being accessible (essential for any govt. service).
It is a neverending source of wonder to me that, whilst our government may be a shambles[0], our government services are so fantastic. They really are very impressive and, for the most part, very easy to use. Clearly a group of people have invested a lot of time and effort into this, and to me they demonstrate very clearly the value of investing heavily in user experience.
Also, the site is almost never down, which is good for somebody like me who tends to remember that they need to do thing X at random-o-clock at night.
[0] Not intended as an especially partisan statement as I have been known to vote Conservative (and over the last few years have criticised both Labour and Conservatives pretty equally), but I think it's tricky to argue that, as an example, sending 3 million kids back to school for 1 day before changing your mind demonstrates competent leadership.
gov.uk is run by software engineers, presumably pretty good ones. They make decisions based on data and know how to interpret it.
One thing that has become super clear to me in 2020 is the huge quality gap between CS culture and public health culture, at least in research. Public health research papers and especially epidemiology papers contain not just severe errors all the time but are flat out deceptive in ways I just never see in the output of CS research or the software industry. For example I never encounter bogus citations in CS papers but that happens all the time in epidemiology papers. Public health research superficially looks and feels like science but when you scratch the surface the scientific method has gone AWOL, good data practices have gone AWOL, basic logic and integrity has gone AWOL. It's just a disaster zone.
They're much worse than any UK private company or UK bank I had to work with.
HMRC is terrible, there are tons of error on the payslips, they mess up tax codes all the times, part of the website uses a previous UI and not the cool modern one, several functions need you to call and queue for 1hr or send letters and hope. Having an accountant helps as they have access to a special version of HMRC they can use to do things mere mortals can't.
Childcare services are embarrassing and require you to have a subaccount in your governments account. I've spent 3 months without access because they asked me security questions for the childcare service on HMRC; I provided HMRC security answers and got locked out; the procedure to restore the account involved verifying my identity by providing via email certified copy of various documents. They were extremely slow in replying (sometimes not replying at all), they never returned any copies or originals I sent and they were extremely picky about certifications so I ended up having to send 3-4 certified and original documents.
The old sites are all over the place.
I think part of the reason gov.uk has been a success is exactly that they were not forced to replace old systems wholesale, but were allowed to build out a platform and methodology that works well and replace public facing services bit by bit.
"How would you architect new features on top of a legacy system implemented by an external agency that is hard to work with and has many bugs but we are unable to change in any way for contractual reasons?"
It isn’t a technical problem...
My approach to this has been to embrace, extend, extinguish (yes, that was an intentional reference, sorry):
Insert a constrained API "around" the service you can't modify. Change all your services to go via that API instead of direct. Now you can apply the fixes you can to the new layer while finding ways to gradually replace it in parallel.
At the same time the new layer provides a constraint: It should be as narrow as you can get away with - creating a "new contract" with the API consumers about what they're actually allowed to do without negotiating with you, and ensure that any expansion of dependencies on the service you can't touch get documented and appropriate attempts made to avoid increasing complexity (it's good to do this in general, but realistically it's easier to get buy-in to enforce in limited cases like this).
This intersection of tech with procedures and politics lie at the heart of a lot of technical architecture work. A huge proportion of the work we do is not technically hard in and of itself - it becomes technically hard because of organisational considerations.
The amount of documentation available is amazing. This is a great example of what's available: https://www.gov.uk/transition . Scroll down to Announcements, hit the first link and you have the full withdrawal agreement in multiple freely open formats.
No it not perfect but it is rather good for this user anecdote.
My overall impression is that we have a rather sensible GOV.UK thingie (not sure of the real brand)
Make sure you settle up before Jan 31 otherwise you'll be fined. HMRC are merciless. Get it done mate.
The pension (Assuming you refer to the private pension scheme) is not filled in and I already know I have to pay some tax on that as I overshoot my allowance.
I normally get it done a couple of weeks before the 31 and doesn't take more than a couple of days even with triple checking everything, so I should be quite in time.
And yes, TBH filling in the tax return has never been a big problem. It would b e nice to automate a few more things (like computing exchange rates for dividends/capital gains instead of having to look them up) but it is fairly straightforward.
They are light years behind countries that have fully-fledged dynamic services provided by the government accessible through the ID Cards, as it is in the Baltics for example.
National ID cards are a particularly thorny issue politically, there's been a few attempts to introduce them, all of which have been noisily shut down in parliament because of privacy concerns. There isn't really any single database of people in the UK, with different ministries having their own databases. In principle this seems like a pretty good approach, even if it does mean that things like authentication with web services is a bit clunky.
https://gds.blog.gov.uk/2020/09/22/introducing-gov-uk-accoun...
I'd put forward Gov.uk/pay and Gov.uk/notify as very successful transactional services that operate on a massive scale. They have been particularly good for local councils who just hook up to Notify and Pay. The parts where security is paramount are handled centrally. It stops councils having to "roll their own".
Universal Credit is a transactional system handling billions in benefits. When the conservative government shut down the economy in March it seems to have coped pretty well:
https://dwpdigital.blog.gov.uk/2020/12/14/dwps-agile-respons...
They have SingPass[1] which is a centralized OAuth for government ID that lets me into things like taxes, identity, etc.
There's a fair argument someplace for too much reach/tracking, but it works really smoothly.
https://www.mom.gov.sg/newsroom/press-releases/2020/0624-pub...
edit:
> The www.gov.sg Portal is the official online communication platform and repository of the Singapore Government, providing you with the latest policy announcements, information and news on Singapore. You can also translate Government terms using our translating tool, and search for contact information of public service agencies.
> The Portal is managed by the Public Communications Division of the Ministry of Communications and Information. We welcome your suggestions for the Portal, to help us to serve you better.
Indeed, it's more a branding website than an administrative tool.
Everything else, they're at the top of the Tower of Blame.
I remember they had a design document ( Something similar to [1] [2], but not the exact one I was looking for ) where they list accessibility as their number one priority and a very long list of accessibility requirement. Simply because Government Web Site needs to be accessible by everyone. That means there are certain fancy things and colour scheme they cant do.
I wonder if the stack for Gov.UK are the same as their Petition Site [3] or was it based on something else? They have an Open Source Github account [4] for Government Digital Service which list everything they have open sourced.
[1] https://www.gov.uk/guidance/government-design-principles
[2] https://design-system.service.gov.uk
You could pretty much spin up the code needed to run the boring bits of a country out of open-source code. How about the application for booking secure prisoner moves? [0] Or some nice open source security guidance? [1]
And that's just one department, most others are similar!
[0] https://github.com/ministryofjustice [1] https://github.com/ministryofjustice/security-guidance
The department overseeing this has guide lines [1] and tests for web pages/apps/forms etc [2]
[1] https://www.uutilsynet.no/wcag-standarden/wcag-sortert-etter...
[2] https://www.uutilsynet.no/regelverk/testprosedyrar-nettstade...
The service manual is an awesome piece of work.
It’s incredibly well thought out and designed.
I've also worked at a company where "as long as the error rate stays under 0.1%, it's fine". As long as each app server doesn't have more than 1 crash per week it's fine.
I can tell you that engineers at the first company end up finding all kinds of super rare bugs, usually in the OS, platform, malloc library, load balancer, etc. They then commit fixes for those bugs which ends up helping the latter company...
I think most places I worked we had no choice but to be in the second company, although I think the rate is a bit below .1%. I'm grateful for all the people hunting down these bugs and making the world better for the rest of us.
I remember around 2006, we still found PHP segfaults that were critical enough I reported at least a dozen. Everything feels so much more stable today (at least on that layer).
That said, I love the wisdom in your comment that you find all sorts of super rare bugs, or conditions that could seriously effect performance, or availability if they become more common (which they often do). Past a point, I've found that an approach which works well is to encourage engineers/operators to drive by metrics, and pay close attention over time to p100's (max), as you've suggested with your 500 errors. Lots of goodies can be hidden behind them, just like you've found with the 500 errors.
You can easily have static content on lots of metals, but the new problem is saturation of peer links on egress, or unintentionally triggering DDoS mitigations along the path that the traffic takes (or on your own or the CDN services).
That the content could be placed on enough metals to serve it is the easy part... but a nicely designed solution for serving the requests isn't as impressive as keeping the network solid and operating smoothly.
This is also why in the tweet thread that performance, optimisation of resources, etc is called out explicitly... fail to do this and you kill your network.
Fastly did great here, but the gov.uk people and GDS also did great in making the job for Fastly a lot easier.
That's not hard to be better in the case of France, half of the procedures require to send some physical letter and the rest are accessible on some atrocious websites.
The UK government however...
https://parliamentlive.tv/event/index/abfe49d3-f24c-4b93-b8e...
Unfortunately the spend control that GDS had, has been removed.
specifically, this would seem to coincide with the PM addressing the nation on TV.
Every time I see this comment about multicast, I think people are missing the point of the internet.
For live events like press conferences or sports this is already solved with broadcasting.
16 million people watched that press conference on BBC TV, broadcasting is very efficent at getting live pictures out. The BBC did experiment with multicast in the past, but CDNs seem to be a more scalable solution ironically.
However in this specific case, the pictures actually left downing street via multicast (SDI form camera into encoder, MPEG-TS over multicast to the studio, back to SDI to get encoded on whichever output chain it is - Terrestial, Satelite, Cable, online)
Indeed the standard to replace SDI (2110) is built around multicast, broadcast certainly uses multicast a fair bit.
[0] https://www.gov.uk/guidance/national-lockdown-stay-at-home
Huge audience plus that equalled massive traffic spike.
Like many, I was hitting F5 repeatedly on this page waiting for a document that didn't seem to be there, eager to know restrictions on various aspects of life. But the /coronavirus page just contained stale advice from December.
Eventually at 8:15pm, somebody found a link to a PDF of the new guidance and posted it on Twitter:
"This has all been done with such crashing urgency that they haven’t even been able to transpose the PDF version of the guidance into the website yet, it is here:"
https://twitter.com/AdamWagner1/status/1346188409300258816
[this is how it played out for me at least, did anyone else see this differently?]
They're the inspiration for many other country's digital teams, including the USDS who are strongly modelled on GDS, and have spawned equivalent teams within other branches of the UK government such as the Ministry of Justice and NHS.
[1] https://www.gov.uk/government/organisations/government-digit...
Also their content security policy in production exposes all their development and staging infrastructure and their AWS service endpoints.
Will you leave it as only a comment or try to fix it by contacting them or a news service?
Not a bad number, I wonder how it compares with a HN hug of death. Sure, caching helps but it's not the whole story (LB, configuring the cache, benchmarking, etc)
Also what dashboard is that?
https://twitter.com/TheRealNooshu/status/1346419468151488512
17M per hour is ~ 4700 per second.
fwiw, the HN hug of death rarely breaks 5-figures for that metric. Its plenty to destroy a wordpress blog on a two core VPS, but its nothing compared to something getting shared in a large facebook group or trending on twitter. News sites are several orders of magnitude larger than any other type of sites besides the mega-platforms (fb) and comms tools like gmail/slack. Its actually a core part of why the entire sector's business model failed together. Everyone chased "reach" (raw audience size) as their most important revenue generating metric (because it directly factored as higher CPMs). Thats how we wound up with dozens of "news" sites all catering to the exact same 100M people clicking awful headlines in facebook shares and then closing the tab to go back to facebook. They (we) solved for scale, but in a way that turned everyone into a commodified copy of each other with no meaningful connection or relationship to the audience. Then facebook just captured all of the ad revenue by gatekeeping/aggregating/"curating".
Being on the front page of HN (or Reddit for that matter) doesn't bring in as much traffic as you would imagine. It's not insignificant but it's not a deluge like what you are picturing. Turns out most people don't bother to click into the article. They head straight for the comments section. Source: work for a company that has been on the front page of both HN and Reddit on many occasions and have seen the traffic.
I only experienced a hug of death once and it was a combination of factors. I had a website on a very particular topic, and this topic made big news (this was unexpected), so I got huge amounts of traffic from every where, (Forums, News sites, social media, and mainly Google searches...). I was down for hours my Apache server just kept dying. One day later it was already old news and my traffic quickly returned to normal.