Comment by jiggawatts | Hacker News Reader

Comment by jiggawatts | Hacker News Reader

You see this in every field, including IT of course.

My favourite example of this was the introduction of the Microsoft Distributed File System (DFS) service into a customer environment. We were only using the DFS Namespaces feature, which is like DNS for file shares: the client downloads a blob of metadata and uses it to find an underlying file share for a logical name. The caching is very effective and the additional delay is negligible.

They had a file performance problem. So of course, they point the finger at the last thing that has changed in the environment: DFS! It must be DFS! I calmly point out that if the performance is bad after a connection has been made, it can't possibly be DFS, it's out of path after the initial negotiation. It's like blaming DNS for slow downloads. Makes no sense! But nope. They just won't accept that. It's the last thing that has changed!

I came back a week later to discover that instead of 2 redundant DFS servers, there are now 6 redundant DFS servers "for performance". To their surprise however, the file shares are still mysteriously behaving poorly.

They dragged me along to a meeting with a cast of thousands to discuss the issues. One guy spent most of the time arguing for increasing the number of DFS servers to 8, 10, or perhaps even more to solve the issue!

Meanwhile I'm having a side-conversation with the storage guy, who sheepishly admits that 3 out of 4 fibre channel paths were down, and this started at about the same time frame as the deployment of DFS. I point this fact out to the room. Everyone looks at me while blinking slowly. A few more moments of silence pass. Then someone helpfully suggests adding more CPU and RAM to the DFS servers. Maybe 8 processors and 64 GB will do the trick!

A few years ago, i worked on a payment processing system for a high street bank. The system, in its first stage of evolution, would receive a file full of payment instructions, parse each one, validate it, then pass it on to a gateway in a special binary format.

An architect had decided we should build this as microservices, coupled by message queues. There was one microservice which split the file into records and parsed each one, one which validated each one, and one which handed off to the gateway.

In the first demo of the system, it took half an hour to process a file with a few thousand instructions in it. This was a ludicrous amount of time: it really should have been done in seconds.

The team met, and one of the senior developers pronounced the solution: since this was a microservice system, we would simply split one of the microservices in two, so we could scale out the amount of hardware we were using. The service selected was the first: rather than splitting and parsing the file in one pass, a first microservice would split the file into chunks, and pass each one onto a second microservice, which would parse them.

I asked where we thought the problem was, why we thought splitting a service would help, how we would measure the improvement, and, moreover, why we thought that adding a network hop would make things faster rather than slower. My questions were rejected as self-evidently unnecessary: it is well-known that you can improve the performance of a microservice system by scaling out.

The service was split. There was another demo. It was just as slow as before.

The project was removed from that team, and transferred to a new team under my leadership.

(this is a true story; the one fact omitted for the sake of drama is that the transfer of the project wasn't due to the embarrassing demo, it was purely down to staffing and logistics)

I don't understand, why do things like this happen? Is someone emotionally invested in upgrading the DFS servers?

They are invested in going down the wrong path. They are invested in not being wrong.

In this particular case it was "politics" in the smallest sense of the word. The head of the storage team did not get along with the head of the infra team, so the infra guys would solve the problems they could solve, ignoring any storage subsystem problems because that was a "dead end".

This is exactly like the boy looking for the coin under the light. It feels like it could be productive, versus definitely not being productive. It doesn't matter that logically it won't work, it's the feeling that matters.

Sometimes it isn't a matter of people getting along. People don't want to see their pet project take the blame for a failure or completely replaced/defunded, Instead of rationally looking at root causes with facts, people may make emotional responses that are dressed up as logical suggestions (e.g. throw more memory and cores at the problem).

Covering others incompetence is breaking a chain.

Reply on news.ycombinator.com