Hacker News Reader

Most of the time, yeah that's how it plays out; it's easy to start from a blank slate. But where I differ is that I'd claim that most early startups don't even have much velocity. They don't actually move with speed, but with haste. All possible corners are cut. As the team scales, the debt comes due and forward progress stalls. What strikes me about this article is the forethought that was put into the engineering from the start - this is a company that really does have velocity and wants to maintain as much of it as possible.

One thing I think can really help is to have two different modes within the same team. You move fast and cut all corners and rapidly iterate when you have an idea by building a prototype, with the only goal being maximizing the speed at which you're learning if/what works and what does. Then you basically throw away all that prototype code and build the feature properly with careful attention to code quality and complexity, testing (++) - since now you actually know what you're building and that it's likely a good idea.

Rinse and repeat, so that you only have prototype code where you're actively learning, to be immediately replaced by properly engineered solutions or wiped away (with the same haste you used to create it initially - no "rarely used, badly engineered" features should stay in your codebase due to priority / resources / attention constraints).

The biggest challenge I've had with this approach has been building and fostering the trust and culture needed to have everyone truly go "all in" for both aspects/modes of working. This breaks down quickly if you have managers or PMs drive engineering to keep the prototype code in for longer, or if engineering insists on not cutting corners during the exploration/prototyping phase...

From https://www.stickyminds.com/aricle/hudsons-bay-start

England's King Charles II chartered the Hudson's Bay Company in 1670. It is considered by many to be the world's oldest continuing commercial enterprise. The company outfitted fur traders for expeditions out of Hudson Bay in Canada. In this part of Canada, referred to as "north of summer," having the right supplies in the correct amounts was literally a matter of survival. And, of course, the company's survival also depended on productive—and completed—expeditions. So the company provisioned the expedition's canoes with the necessary supplies for the trip. Then they sent the expedition team a short distance in their canoes to camp overnight. This was a test. The first camp was merely a "pull out," commonly called for many years a "Hudson's Bay Start" (HBS). Said Sir Samuel Benfield Steele in 1874, "[This was] very necessary so that before finally launching into the unknown one could see that nothing has been forgotten, or that if one had taken too much, being so near to the base, the mistake could be easily corrected." This risk reduction technique was not free. It cost the team precious time in a short trading season. They had shipping deadlines to meet. I can imagine them saying "Yes, but we really have to be on our way or we'll miss our shipping schedule."

Interesting and informative, thanks for sharing! Looks like your link lost a ‘t’ from the word “article”

This sounds great on paper but I have a hard enough time getting buy-in to finish the prototype. I've never seen anyone throw away the prototype and rebuild from the ground up before waiting years and then calling it "V2"

An alternative way to deal with an unreliable prototype is to make a reliable wrapper around it.

Such as paying time in exchange for reliability (e.g. TCP/IP, retry and backoff), disclaimer for unreliable behavior and law-related issues, version gating for file formats/protocols, redundant storages, and killswitches for going rogue-prone machineries.

> They don't actually move with speed, but with haste. All possible corners are cut.

This sentence strikes me at my trauma center.

The company I'm in is at the critical phase of startup. 4 years down the road of hypergrowth, the knowledge gap and quality gap is apparent. But then I realize that corners are cut because of a lack of expert in the area whose authority is recognized.

Most experts and brilliant minds have the same idea about the corner cutting. When foundations are designed to be reliable, velocity may jump exponentially, they know. Reliable isolation, version gating, and enablement over service is how they should move, they know. Product design and physical constraint has at least one coupling string, they know.

But these people aren't given the authority, not invited into the room where the decisions are made. In the end, those who bring the immediate money prevail over those who bring mere potential of 10x more money.

As the article said, hire the best engineers. But what the article miss is to hire the best engineers who is invested, can thrive in the politics of startups and use the attained power to bring everyone to success.

If companies took the concept of an 'error budget' (every SRE principle seems to be important to companies except this one) seriously and saw it as a signal instead of an annoyance there would be some ebb and flow in this realm instead of a continued compounding increase where stability sits on the backs of a few people with the tribal knowledge. Just my 2 cents.

Say more, please.

How does an error budget account for a lack of experts in the stability area? I can't seem to be able to make that connection, and I'd love to learn.

I'm guessing the suggestion is: once the error budget is spent, no new feature work is accepted until causes of the errors are fixed

"Fixing the error" is at times subjective. A less nuanced approach is to freeze non-reliability-improving changes (i.e. merges) until the production meets SLO again. That is the canonical example policy given in Google's SRE Book.

Sure, that's solid policy, but I'm not sure it really addresses the "tribal knowledge" issue per se.

I mean, I'd like to think that fixing reliability issues increases knowledge, but decades on multithreaded codebases littered with "sleep(0); // just in case" point in a different direction.

All I've found to work so far is a deliberate attempt to shift to more active knowledge sharing. I was hoping to learn a few new tricks from OP, but "reliability freezes" are not it, based on my experience.

I agree that you can't maintain the same velocity you started with as the company grows, but there are definitely ways to keep as much of it as you can. Some companies slow down a lot more than others.

In my experience, it ultimately comes down to SDLC. Teams need to be able to work (mostly) independently of each other. This means being strict about using API boundaries rather than sharing database tables between multiple apps, and avoiding breaking API changes that force you to deploy software written by multiple teams in lockstep. That is the real reason microservices are valuable: they decouple teams.

To use Frederick Brooks's term, the optimizations, the special cases, the interactions with other parts of the system, start to become essential complexities as the system grows.

The skill of your developers is all that stands between the code base and the total chaos of insurmountable technical debt.

Only to the extent that presence of mind and ethics are skills instead of something you are. People don’t clever their way out of these holes. That requires wisdom, not cleverness.

If you treat performance as a feature, then you can do a lot better by this yardstick than most. The problem is that so many people don’t know what that feels like that they fight you on it. I’ve heard stories about managers trying to dismantle processes I “forced” onto people only to find out they didn’t want to give them up. They just didn’t understand why I was so vehement that we have them, until they had them.

Most people assume that suffering is the natural order of things instead of the default state. So it doesn’t really track for them that choosing your form of suffering today can limit the amount you’re doing later.

> Startups are fun to work for early on, when complications like these don't exist. It requires more than "the best engineers" to keep up development velocity, let alone feature velocity.

This blog post (which is pretty high quality for a content-marketing piece) talks about exactly these issues: https://archive.is/FQKJH (athenian.com), Scaling Your Team From 5 to 250 Engineers.

I completely agree that it is not possible to maintain the same velocity. I have always optimized to decrease velocity as little as possible as you scale the team. That you really want to avoid is doubling your team and your velocity dropping by 50% or more.

Ideally you double you team and velocity decreases by 10-15%.

They started with "best practices" that maybe slowed them down at the start, but surely made the scaling smoother.

Of course scale will have some effect, but the recommendations seem very sensible like keeping teams small and independent.

When we started in 2009, we did 3 major releases a year (enterprise software, deep vertical SaaS).

We now do monthly releases into production, with 80+% marketshare and 500mil rev just in my product line.

Scaling is hard, but doable. Takes a lot of willpower and fighting against gravity.