Comment by dimitar | Hacker News Reader

Comment by dimitar | Hacker News Reader

https://www.tsb.co.uk/news-releases/slaughter-and-may/ - The bank has commissioned an independent report

https://www.tsb.co.uk/news-releases/slaughter-and-may/slaugh... - the report itself

Interesting notes from the summary:

* Functional testing took 17 months, and began very late due to project delays.

* The board was told that there were only 800 bugs, and yet the independent review found that there were more like 2000

* Non-functional testing seems to be rushed

* the Sabadell COO reported SABIS, one of its child companies and major IT contractor in the migration doesn't have the capacity to respond and solve incidents after go-live

And the TSB board decided to Go Live anyway.

Looks like a classic IT failure story - project is late, save costs on operations, get tired of testing and fixing and decide that it is good enough to be released. But also some very weird incentives - SABIS was apparently not the best vendor for the job, but they are a part of Sabadell, the bank that bough TSB.

Looking at the "Integrated Master Plan" Gantt chart of the project on page 44 is Waterfall as hell and they still failed to plan for Non-functional testing.

These some first impressions, it looks like a interesting read.

In UK/EU banking the way to get promoted is to be a yes man. Business has complete control and abhors IT. Impossible deadlines and hands-off approach make atrocious code bases. And this becomes a never ending feedback loop going on for decades. There’s never time to pay for technical debt. And now with the replacement of commercial staff for IT staff there’s a growing divide. Stay away.

I have experience of this and can confirm. Banking is not a sector for young smart people because it will wear you down. It's a constant battle of change approval boards, fighting their risk adverse fear of causing service distribution while at the same time expecting things to be delivered. Save these types of roles for when you are old and want a boring job.

Unfortunately, the "finance is only for old and boring people" also leads to a certain prestige for finance - startups are for kids, finance is for adults. Many in finance almost pride themselves on how much time they spend in meetings, and how long it takes to release anything.

For startups, the myth is that success comes from a "move fast and break things" culture, so they wear IT disasters with a badge of honour. In finance, it's the opposite - with enough planning and governance, huge budgets, and long timelines, they believe you can guarantee success on any project.

The culture is don't make changes at any cost. As a result you don't make many changes, and as a result you're bad at them.

Modern technology companies make tons of changes all of the time. As a result they become good at them.

Looking at the reports they learned EXACTLY the wrong lessons. It's not to make fewer changes, it's to become good at changes.

It's also the risk distribution of one large vs. many small changes that heavily favors many small changes.

> Modern technology companies make tons of changes all of the time. As a result they become good at them.

Really? That hasn't been my experience. Nearly every time some moderately large update to any modern software goes out it seems to break a bunch things. What it doesn't break it usually just ditches entirely.

The trick is not to have large updates...

Move a bit at time.

Modern tech companies can't even seem to do relatively small updates without fucking them up. Just look at Windows Update for fuck sake.

I think modern developers just make a whole lot of bullshit excuses for why they are crap at their job.

Those generally aren't what are considered to be small updates. Organizations doing small updates to are doing them multiple times a day. (e.g. FANGs)

My experience is those is that those lessons (e.g. the need for extensive automatic testing) are learned the hard way.

> small updates > Just look at Windows Update for fuck sake.

Yup, those updates that change the operating system that is run on billion of devices in extremely varying hardware and configurations. Very simple.

It’s the same in the US Federal government IT space, especially when dealing with archaic legacy systems. Innovation is nearly impossible and change approval boards reign. Everyone is risk-averse because of previous breaches, even to the point where engineers are not allowed to run the code they wrote themselves. You heard that right. We can write code, but we can’t run it ourselves.

How would they even stop you from doing that? It just sounds like a challenge for a workaround to me.

Had similar experience talking to ... someone at fidelity. I can't remember his title, but we were trying to negotiate some data exchange with them. He asked for our development workflow/process, and balked when he saw that a "developer" would also be the same person who had "access" to the database with customer information.

"Developers can not touch the database"

Umm... someone needs to, and we're a small team.

"Developers can not touch the database"

"Why not?"

"They can exfiltrate data".

"So could someone else who has access to the database"

"Developers can not touch the database"

"If we give a non-developer access to the database, by definition, they have access, and therefore the ability to exfiltrate data"

"Developers can not touch the database"

"Can you provide some examples of how other companies who've passed your inspection are making any sort of database changes/upgrades without someone having access to the database?"

"Developers can not touch the database"

And that ended it apparently.

This is a pretty normal requirement. I believe what they are saying is that you should not be able to access sensitive customer data by virtue of “just” being a developer, and you should audit who has access to such data and why. And by “touch” they mean unfiltered access, not “you cannot do anything with the database and therefore cannot work”.

This is pretty basic information hygiene, and if you don’t have enough care to implement even that then why should they partner with you? Sounds like a liability waiting to happen.

> This is pretty basic information hygiene

They would not give us any indication of what they expected.

> I believe what they are saying is that you should not be able to access sensitive customer data by virtue of “just” being a developer, and you should audit who has access to such data and why.

It should be pretty straightforward for them to provide a "here's our initial base minimum requirements guidelines" and work from there.

The parameters they laid out were similar to "anyone who would be capable of running any code on the database should not have access to any data". Every clarifying question about "what about X? what about Y?" was met with basically, "no, that person could exfiltrate the data". Which, yes, anyone who has access to the data could, theoretically, exfiltrate the data.

I showed a demonstration (from laptop) and the guy got really upset that I had 'customer data' on the screen and in my database. It took several minutes of back and forth to explain to him that it was "faked" data.

It sounds like your team has no clue how to approach these discussions and lost out on the partnership. Fidelity don’t know you from Adam, and you seemingly gave no impressions that you had any idea what they wanted or how this process works.

They shouldn’t have to spell out how to protect data to you, and simply by asking “what about X” as if you are the only one to ever say this to them or notice that there are of course ways data could be exfiltrated betrayed your ignorance, not theirs.

Did you expect them to say “my god! Of course! Why didn’t I see this before? This regulation we are legally and contractually bound to obey is not perfect, so hey why don’t you just ignore it.”?

And to top it off you started showing realistic looking data without being upfront and clear to them that it was synthetic? Come on.

Not the same as "we're doing this and... " being cut off and being told "this is 100% wrong, and we will not tell you any scenarios that will bring you in to compliance with our view of secure".

We had plenty of secure process in place. The moment he heard the word "developer" and "database", he basically shut down and refused any meaningful engagement.

It is not his job to educate on secure practices. It was his job to inspect what we had in place, and he stopped doing that the moment he heard those two words.

"Would having a non-developer administrator connect and process the changes?"

"Developers can not have any access to any production data".

That's not a meaningful response to the question.

> And to top it off you started showing realistic looking data without being upfront and clear to them that it was synthetic? Come on.

I'm not sure how much more 'up front' you can be besides saying "we develop with faked sample data, this is faked and unrelated to or connected with the production system" at the start of that walkthrough. Perhaps if we'd opened up a Northwind database it might have looked more plausible?

No one was expecting them to ignore anything. We were expecting someone to ask "are you doing X? how are you doing Y?", then followup with "you need to remediate these aspects of your system before we proceed further". That's what I would expect.

That's because the person you talked to probably did not know, either. They were basically reading off a checklist.

To be fair, it's not their job to educate you on good security practices. This was an obvious red flag (developers should indeed not have unfettered access to production databases), so if you fail at a basic separation of duties on the team, there's no point in progressing any further. If you're expecting some "detailed guidelines" so you can "fix" your system to pass their audit, you're missing the point entirely.

Also when showing anything that remotely looks like customer PII, it's your job to clarify that this is not real data. Given the precedent, I can see why they thought you were being careless and showing real production data.

Wait a minute. How would they be showing production data if they couldn't access the database were, presumably, this data was stored?

It sounds a lot more like the smaller company was interfacing with someone who was not technical or at the very least did not have a clear understanding of the internal processes of the big company, and expected the smaller company to have prior experience of this thing and to guide them through it.

Most likely, the process would have involved the samaller company communicating with someone inside the larger company, like a DBA, and asking the large company DBA to make the changes they needed. In large financial corps, company developers are not allowed to touch the database either, in fact where I worked, we were not even allowed to create a database for testing an internal-facing project. Processes are that anal.

> How would they be showing production data if they couldn't access the database were, presumably, this data was stored?

The entire anecdote was about them telling the representative that they DO in fact have access to the production DB. Why would you expect him to act as if they didn't have access after being told they do?

Oh, my mistake- sorry, I misread that.

I don't know how to fulfill this that makes every auditor happy, but so far what we're doing is, we have a restricted-access role on the db that denies access to certain sensitive tables, create a time-restricted temporary user for the developer, and then give them access to the read-replica.

This sounds like the right approach, and it’s similar to other companies I’ve worked at. The time limited nature is nice and presumably there is a audit log of some kind tied to that.

In large companies, especially large companies that are not technology companies, IT people have very clearly delineated roles, so a "developer" and a "DBA" are different people and when a "developer" needs a database, he or she must ask a "DBA" to make one for them.

In small companies, a "developer" is, by comparison anyway, a jack of all trades, who is probably expected and required to know how to do every possible job, including database management. So when the large company said "no developer" they possibly meant that, internally, their developers could not touch databases and only DBAs were allowed to, and they expected the smaller company to have similar processes in place. Which is quite unrealistic.

At the end of the day, paranoid as that may sound, the big company person was probably getting stuck on the role description of "developer" and would not have freaked out if the smaller company had used the term "DBA".

No no no, this is where you request a larger budget, change everyone's title to DBA, and have DBAs write the code (but phrase "code" as "queries")

While I get the 'atmosphere' of the meeting just by reading this, there are quite a few valid points raised. Developer should not have access to real user data. Devs should be provided with DB schema definitions,info on data volumes and any other relevant information or metadata that is required to develop new features/migrate/etc. When it comes to databases with extremely sensitive data,i.e. banking, healthcare, criminal records, there must be an additional layer security that'd log anyone accessing particular records.

Separation of duties regulation. Nothing wrong with it. The idea is to have a wall so that you can't have one person able to write and push code to production.

Now if you mean "run code" at all that's another story.

Unfortunately in a lot of countries in Europe that's the only sector that does pay a premium in comparison to other sectors. Try getting £100K+ dev job in London and suddenly all the potential employers ( with a few exceptions) end up being either in the City or Canary Wharf.

My theory is that the code never gets fixed in these pathological companies. They just get higher costs until people switch away to a bank that is more efficient.

You could replace banking with any other large enterprise vertical like healthcare, telecom, government and be very correct. The problem is that IT is a cost center, and cost centers are as a rule terrible to work in regardless of industry, even if you're at a tech company.

Very true. Work somewhere where the CEO has some level of understanding and appreciation for your work and life will be much better.

Corporate IT is an especially bad place. You can only lose. When things go well they start cutting costs by letting people go and offshoring jobs. And when things go badly you get the blame. The IT department at my company is hard to work with and often drive me crazy. But when I look at their structure it’s obvious that it doesn’t allow for people to a good job.

I think that’s especially common in regulated industries. On the one hand top management are afraid of compliance issues but on the other hand they don’t understand and respect why things cost money and time. So they are very susceptible to hiring sweet talking people who pretend to be able to solve this.

> * The board was told that there were only 800 bugs, and yet the independent review found that there were more like 2000

If there were only 2000 bugs in a full bank migration, they were doing pretty well.

Our fairly small single page app is going on 300 now after some 5 weeks of testing.

It sounds to me like there were 2000 reported bugs. You likely had better coverage due to smaller scope.

The number alone is meaningless. Some bugs are UX inconveniences, while others can destroy data, as we saw in this case.

The real problem at these banks is the multitude of systems and datafeeds that impact systems downstream a hundred systems long.

The real solution to these big project failures is, in my experience, the phase deployments. With the first step being retiring old systems and consolidating data repositories to reduce knock-on effects.

I thought the same, but then figured the ratio is probably around the same.

Reply on news.ycombinator.com