Hacker News Reader

What? Facebook, Amazon and Google do not have external users?

"We need to be careful because customers use our software" is a pretty lame excuse. Every company needs to properly test, document, package their software. Some companies do it better than others.

Where they have external paying users, they still operate the software for them (i.e. provide it as a service), so they have access to all what these customers are doing.

It's not a lame excuse, unfortunately, there is a very real barrier of trust when you talk to a coworker (even from another group) and when you talk to a customer. At the very least, you can agree on a common schedule.

IMHO it's not a coincidence that the blog listed those companies as more "nimble".

He also hints that his customers might have more power to push back on changes. That's certainly the case for smaller software houses that sell to big enterprise customers.

Facebook, Google, etc, are somewhat beholden to end users, but not in a way that impedes them from rolling out what they want, when they want to.

Good point. The worst-performing companies I've worked for had one thing in common: they were small and unable or unwilling to push back on toxic customers.

Pushing back doesn't always work. A good example...support for IE6, way past the time when that was a good idea.

Your contact at "big mega corp" that pays for your software may have no ability to change their internal policy that they've stuck with IE6 for years after it was sunset. So, if you want the sale or renewal, you have to support it.

Knowing your customer lets you take shortcuts. For example, if you have a monorepo, you can do a code search to find all usages of a function and fix them, instead of going through a formal deprecation process.

There is a tension here between efficiency and privacy. Inside a corporation with an "open" culture, you can be a lot more transparent and efficient since you don't have the same privacy issues.

Open source projects can sometimes do similar things, for example with the Linux policy of having device drivers in-tree.

I think with customers he meant mainly customers who customize and extend the software. Especially if you publish an API and customers can run the software on their own servers you are kind of stuck because you don't know what is being used and how. With cloud based stuff it's a little easier because you can track what people are doing.

Also, Facebook has a history of breaking things for 3rd developers. You can't really do this with enterprise customers who pay a lot of money for your stuff.

Yeah, I worked for a startup that grew large quickly through a drive to get innovations to market fast and fix bugs later, and what slowed us down was winning enough large contracts with enterprise companies and governments who wanted more and more red tape and documentation and guaranteed processes, and basically exported their risk-averse culture to us.

We worked around this by having separate release channels, one for our enterprisey customers who opposed all change, and one for our customers who just wanted the coolest things first and were willing to accept some rough edges and lack of documentation.

That doesn't work if _all_ your customers are risk-averse, of course.

Developers always have to answer to someone. That might be a client who's buying bespoke software, or a customer who bought something off the shelf, or a user who uses something that's made internally within a company. If there's someone who uses the application then there's someone who you have to justify things to. No developer who writes code for money can just push out changes without worrying about breaking things.

The view that people who build and sell products can go faster and have less stress than people who write code for other people is nonsense. The pressures basically are the same; they're just from different groups of people.

> We don't know what things we can break or deprecate, so we have to be extra careful.

I'd argue that almost nobody really knows that. [1] It's just that many developers / companies somehow get away with not being extra careful. How many libraries introduce some "unimportant" backwards incompatible change without bumping their major version number?

[1] ... unless you have a web application and really good statistics. But then, you end up being a large company like GitHub or Amazon, I guess.

Thinking about it some more, conservative users are also a factor. If we change anything, we impose additional cost on our customers, which have to retest, redeploy and retrain. So it's not easy for this reason alone.

When you have a platform for internal use only (or a platform that is available free of charge without any warranties), you can decide to make a change by fiat.

We would like to "move fast and break things", but our customers unfortunately don't appreciate that sentiment.

Thinking even more about it, many changes are "cosmetic", such as renaming fields or restructuring essentially flat structures. These may make the system more consistent and easier to understand for newcomers, but break old code for "no reason".

So developers should develop some kind of taste to distinguish between "real" changes and "cosmetic" changes.

You're trying to use Google, Amazon, Facebook, as an example of a company that has no external customers and takes no risk when deploying untested features???

You must be kidding.

It's actually quite apt. AWS is a different beast, but Google and Facebook can and routinely do ship broken code to production that drops queries on the floor, serves broken JavaScript, redesigns the UI in ways that confuse millions of people, etc. It doesn't matter because the products are free. The worst that can happen is you lose money by failing to serve ads for a while.

Selling into the enterprise space can be very different. There even a single dropped or bogus query can have a big impact on the business. They are paying, so they get to decide if and when they upgrade. You have a small number of very important customers, instead of a billion unimportant customers, so losing one because you didn't meet their bizarre requirement is a big deal, whereas in the consumer space if you decide your app is iOS only, OK, no problem, you can still be successful.

Reminds me of this: https://xkcd.com/1172/

I made updates to some of my code security reasons that ended up speeding things up a little bit (smaller binary to load + less reliance on permanent storage). This caused other developer's code to crash because he inadvertently needed my code to run slow under certain conditions (it wasn't obvious that the issue was a speed thing).

How do issues like this happen? I'm fairly new to software development and I haven't really seen issues like this, and I'm having trouble thinking of what types of code would cause it.

I'm assuming it's usually related to things using multiple threads or at the boundaries of different programs communicating?

Yes of course. It's concurrency related. You write a program that starts something, does some work, then tries to access the something expecting that it's already started. On your computer, the "does some work" stage happens to take long enough that the thing has indeed already started. On some other machine, it doesn't and you get a crash or hang.

It happens quite often when developing concurrent software.

> Companies that do not have external customers, but write software only for their internal use [...] do not face these problems. They know who their customers are (guys from the other cubicle), what features they use. The can take a risk of deploying an improperly tested feature.

In a non-software company, this is not my experience at all.

While some end-users are local, all communication must route through distorting managerial layers. Huge swaths of features exist from fearful inertia.

> Facebook, Amazon or Google, do not face these problems

Or eventually they do as internally everything starts to rely on everything else and they can't move fast enough when competition arrives.