TSB Bank: Botched upgrade has left customers unable to access their accounts
theguardian.com
theguardian.com
http://prensa.bancsabadell.com/en/News/2018/04/banco-sabadel...
Backup 1: http://archive.is/WnwiB
Backup 2: http://webcache.googleusercontent.com/search?q=cache%3Ahttp%...
Funny and poignant
In my current job I have repeatedly told non technical superiors that reverse engineering an outdated API is not a proper solution and could fall apart at any time. They do not care. It doesn't even register. Do civil engineers have this problem if they realise there might be a structural weakness?
This could involve going to the press, to a standards body in the country, to the government, to their union or industry representative body.
In the UK there are laws protecting whistleblowers who take actions like this.
As a PE, they can face significant legal repercussions for errors in work they approve.
The benefit to the PE is that many Civil Engineering projects are legally required to have a PE design/approve them. The PE literally has a physical stamp they use to stamp blueprints and designs when they approve it. Work won't proceed without that stamp.
So, if a PE refused to approve a bridge design, presumably it would be very difficult to find another PE who would approve it (especially given the record of the first PE rejecting the plans) and the work would not proceed.
I think that's a bit silly. The vast majority of software is trivial stuff and no real harm is caused when something goes sideways. Few of us work on safety critical systems. When a lawyer screws up it often does a lot of harm and they are also inherently bound by our legal systems. Corruption and negligence/incompetence in this area lessens the public's faith in those systems.
Lawyers and civil engineer inherently take on far greater responsibility than your average Joe Schmoe programmer writing yet another CRUD app.
Yeah I can get behind that. I work in biotech/cancer diagnostics/prognostics and we could definitely afford a higher standard, especially around the core stuff (e.g. the CV/algorithms people, but also the typical IT end of things.)
If you pass, you get a stamp, which lets you approve drawings as a certified PE.
If you believe something is not designed to adequate standards, as a PE it is on you to not put your seal on the drawings. They can't build the bridge without an engineer doing this.
If you're taking about options when some other engineer has already approved the design, I'm not sure what the best course is. Maybe bring your concerns to the engineer in question and work from there.
Or if the design is fine but a contractor is cutting corners in construction and not building to the specifications, you could potentially engineer a new design to reinforce the problems, or you could make them take it out and build it right.
(Context: I am the Head of Engineering at a bank.)
And I'm not even talking about Facebook, more about things like cloud bleed. How do you even try to regulate something like this? It's not like they intended it, but they still majorly screwed up.
This appears to be an IT failure comparable to the 2012 and 2015 IT failures at the Royal Bank of Scotland. These involved multi-day / multi-week full/partial outages, and multi-million pound fines.
FT: https://www.ft.com/content/9178ef60-46d9-11e8-8ee8-cae73aab7....
Background: TSB is a UK bank. They had a planned 50-hour maintenance window over the weekend (Fri 16:00 - Sun 18:00), to move away from Lloyds's systems (following the "un-merger" in 2013-2014). This went wrong, in what is likely to become a textbook example of an IT failure.
From a letter from Nicky Morgan, chair of the UK Parliament's Treasury committee, to TSB CEO Paul Pester:
> The reports of unauthorised transactions, access to other customers’ accounts, and failures of in-branch services have all the hallmarks of an IT meltdown. This is yet another addition to the litany of failures of banking IT systems. Potentially millions of customers could be affected by uncertainty and disruption.
Some interesting screenshots of people seeing debugging information in the front-end: https://twitter.com/bcleeve/status/988333991676796928 https://twitter.com/apphancer/status/988698182170357760 https://twitter.com/thejackthomson_/status/98856435451268710...
Ironically, TSB's parent company issued a press release (in error?) claiming that the migration has been a "success"! http://press.bancsabadell.com/en/News/2018/04/banco-sabadell...
Sabadell warned about ‘high risks’ in IT integration of TSB https://www.ft.com/content/c5157c1e-20ab-11e5-aa5a-398b2169c...
Really? I'm with TSB since 2011 and it never worked nice for me. When I login I always get
Greetings, ???NAME??
Based on the original planned 50-hour outage, I suspect that TSB chose a "big bang" migration: all-or-nothing, no rollback possible. These are technically easier (so cheaper) to develop, but far far riskier than taking a phased approach.
I have successfully argued against such approaches in the past, due to the high risk of catastrophic failure if it goes wrong (i.e., exactly what has happened to TSB).
I regard TSB's failure as an in-the-making textbook example of "how not to do it".
I remember one migration which involved lots of internal services which were all tightly coupled, meaning updates across the whole backend, not just the part that needed it and no way to do a phased rollout. It was made worse by the fact that the update was to the platform and that depended on a DB upgrade, but that upgrade was incompatible with the old version which meant everything had to be done at once.
They had no disaster recovery plan. They got it done, but I'm sure a lot of people involved went grey early thanks to that nightmare.
Of course the business had no idea of the utter mess that caused all of this and continued to make harmful decisions in the name of saving money and further underfunding the IT department.
In the specific case of TSB, I have my suspicions about the reasons for cutting corners in this migration, given: https://www.thetimes.co.uk/article/missed-deadlines-leave-1m...
Tech support switched me over to a part of the site that had not been updated, and that worked for a while. Then it didn't. Back to the bank I go, with a new customer service representative, who really didn't grok the issues (new to the company?), and who became quite snippy.
Very aggravating.
Still no access to my old account this morning...
Apart from that they're both much the same, and both leagues better than a normal bank. It's amazing buying stuff or getting cash out and the notification comes through on the phone before the ATM tells you the cash is coming!
Another good thing both have - you can switch off cards instantly. I thought i'd lost my Starling card when out last week, instead of cancelling it I just temporarily halted it until it was found.
What I do love about Monzo is their approach to development - they blog everything, even the outages. Good to see the process involved in making a new bank from scratch. I'm keeping the account as a scratch account but have already moved the direct debits from my old Halifax account to Starling.
I am most definitely joining them (but keeping my "main" bank)
As for the TSB, it would be interesting to see what went wrong and how they fixed it.
This seems to be a very common pattern, and I wonder if they are having issues because of it. I was a Monzo early adopter and even I haven't moved over entirely.
I think a lot of users _liked_ the prepaid cards, to help them budget spending money, etc. I suspect that a lot of people haven't upgraded to a full bank account with them as a result.
Basically you can put money in your account into separately named isolated areas in your account that can't be spent using your card normally. In layman's terms as many saving accounts as you want with more flexibility.
You can still kind of use it as a prepaid card by just topping it up from your bank when you want. It's just better as you can now use Android Pay etc. and have proper banking functions like having an actual sort number/account number, direct debits, standing orders etc.
My Amex is linked and I get a notification of all purchases as they go through regardless of whether I use my phone or card.
If it fails because of a failed upgrade, hacker attack, AWS not being available, etc... what is your plan B ?
EDIT: Sorry correction, looks like they sold the stake in 2015.