Things I learned from building a production database
maheshba.bitbucket.io
maheshba.bitbucket.io
> [17] For storage systems, bias heavily in the beginning towards consistency and durability rather than availability; these are harder to measure and harder to fix if broken. Because availability is easier to measure, there will be external pressure to prioritize it first; push back.
It's kind of crazy how well the author is able to move from the very top to the very bottom of the technical project management stack.
But don’t you think the fact that our organizations require someone to be “full stack” organizationally to be a good tech lead is a failure of itself?
I wish organizations can support those talented engineers even though they might not have the best “people/soft skills”.
> We hit production with a 3-person team in less than a year; and subsequently scaled the team to 30+ engineers spanning multiple sub-teams
Starting with a small skilled team to build the core and actively plan for a larger team at some point is absolutely critical for project success in my experience, especially with complex projects like this. That small team can have a much tighter communication loop and all keep a single unified picture of what the end goal is. When the team grows this single unified picture is no longer feasible so it has to be broken down into sections, which is a good thing, but if the core isn't setup so that the sections are sufficiently decoupled things fall apart quickly.
And I totally understand the impulse to kill it. Do not unleash it onto the world unless you have thought about your projects long term future (something php & js ignored to do at the time it seems). Thus you end up with tons of pet projects that are either sensibly kept private/ununleashed or because people fear the judgement of others (aka the kill it impulse).
I'd say the nature of the beast is that entrenchment/reinforcement of existing tools/patterns and mental models tend to stick around as they are safe, so a ton of innovation never occurs as someone might not bother to follow a certain path. It's the same with water flowing into existing grooves/cracks in rocks, instead of making new grooves, unless forced to.
When new platform is being worked upon it's extremely difficult to get product teams to adopt. The PMs on the product teams view a new platform as a distraction because it seemingly doesn't help them launch new features. Worse it's taking the engineers who could be working on features to integrate with this new platform.
Typically during planning phase not too many oppose a new platform. However, when it comes to integrating (or adopting it) then everyone suddenly becomes conservative; no one wants to be the first mover, take the risk to be early adopter. It could also stem from product teams' prior experience of having burnt their fingers trying to integrate with a new platform.
I have seen quite a few platform teams get abandoned after everyone signed up. The platform team need to put in an extra effort to get product teams adopt.
So from that perspective you're describing the first two "explore" and "expand". The third is also interesting "extract" where a project reduces risk over everything else.
Both are missing the fourth part of a lifecycle. If we take the 4x analogy further (which is a popular video game genre) then there would be a "extinguish" phase. In a 4x video game that is where you take over the whole play area (map etc.), essentially by bullying out or assimilating the other factions.
In the real world this is also kind of true (see Amazon as a popular example in tech, or price dumping strategies practiced by Pharma and other large industries). But it is generally frowned upon as it hurts the economy and erodes social contracts. Also for most projects and organizations it isn't feasible to do so. So the "extinguish" part could be understood as "gracefully fading out" from which new cycles can be grown.
[0] https://medium.com/@kentbeck_7670/fast-slow-in-3x-explore-ex...
> Let us first examine the tendency to overpopulate a design effort. It is a natural temptation of the initial designer-the one whose preliminary design concepts influence the organization of the design effort-to delegate tasks when the apparent complexity of the system approaches his limits of comprehension. This is the turning point in the course of the design. Either he struggles to reduce the system to comprehensibility and wins, or else he loses control of it. The outcome is almost predictable if there is schedule pressure and a budget to be managed.
> [26] In a transparent codebase with quick review cycles, APIs will leak implementation details unless you gate-keep.
It's tempting to make quick fixes to APIs but please take it slow. Think through existing customers as well as how could new ones will use it.
Also, something I learned while building/maintaining public APIs was to always clearly document (along with examples) the context in which an API should (ideally) be used and how it combines with other APIs to enable common use cases.
so by that learning it is not at all weird that they would refer to infra at an alternative company
This is pithy. I'm curious - does anyone here follow this philosophy? How do you actually design as a team? A meeting for every single design decision?
I'm used to the team finding general alignment on the problem in meetings, but then a single person coming up with the design, writing a doc, and circulating it for comments.
Which is understandable, as you have to integrate anyway and doing it later is more painful than doing it earlier.
Concretely, let's say I'm designing a term-rewriting system. The program will be inspecting an abstract syntax tree (AST) for certain patterns, and if it detects those patterns, rewriting them. (e.g. if we see the AST for "1 + 1" rewrite to "2"). I might write up a document on the architecture for the system, and a list of rewriting rules I need. How those rules are written can be left up to individuals on my team as long as it fits into the architecture, has the required functionality, and is of reasonable quality.
On a smaller scale I've worked in teams where when someone picked up a story, for certain cases we would first do a rough outline of the implementation and then take that to one or two devs from the team, just to talk about how to tackle the problem. Not a meeting, just five minutes of "here's what I think would work". Its hard but good alignment before writing code really helps with finding the errors in a design.
Once we have done that, everyone is aligned and it doesn't matter who to implement it; it will be implemented in the same way.
Something tells me the author deliberately stopped at 42nd item.
[22] Avoid using real-time for correctness guarantees or comparing clocks across machines unless you have (and understand) error bounds on the clock.
I shudder to think of the hours spent head-scratching that went into this Lesson Learned :)
https://static.googleusercontent.com/media/research.google.c...
It goes into a lot of detail on this
Edit: Also intelligence is an over-rated metric, persistence is far more important when it comes to these hard problems.
It's very easy to get lost in day to day mess.
I also think we might be underestimating how much one could do with one year, a relatively well-specified problem, and a team of three super motivated and intelligent coders.
[2] Be careful to have the right number of customers (in the beginning, just one) and the right customers (whose requirements allow you to build out key technology); and grow that number carefully.For example:
> Measurement is a means, not an end.
Reminds me of a lesson I learned early in my career. Bad things happen when you lose sight of your purpose and focus just on measurement.
I've never actually thought about doing this as an initial design principle. I wonder how it materially affects the design.
In our case, this design principle mostly applied to our internal APIs. Particularly, we designed our internal sub components (consensus protocol, particularly) in a way we can swap it without any downtime.
"[4] But realize that customers may not express what they really need; don’t take requirements at face-value, instead spend the time to understand their use case in detail. Read their code." - Yep - Users Are Liars. They will lie right to your face, about requirements, deadlines, use cases, functionality, costs, etc, for no reason at all. Never believe anything a user says. And obviously, never tell them you think this.
"[9] If you get good and/or aligned managers, be as understanding, supportive, and accommodating as you can. If you don’t get such managers…" - Become the shadow manager; Team Lead, Lead Developer, Delegate Architect, etc. Ask the manager if you can get "hands-on experience" with various tasks like: running ceremonies, taking meeting notes, coordinating work, planning, resolving blockers. Reach out to other teams in private to resolve problems without involving any management, and be careful that the people you reach out to don't widen scope to their managers. Hide problems and bubble up accomplishments to the bad manager. You'll get none of the credit, but the manager won't have the opportunity to make everyone's lives bad.
"[10] Make your project robust to re-orgs." - Of these, the most resilient I've seen are the ones that promise the moon, and then deftly delay everyone's realization that it doesn't live up to the hype until they're all deeply bought in. Don't make the mistake of trying to get an executive to buy in, because they'll be gone in 3 years and everyone will ditch your project.
"[15] Design APIs with migration to new implementations as a first-class consideration" - Yes, yes, yes!! Literally all technology is replaced eventually, most of it in 3-5 years. Plan the exit strategy at the outset.
"[21] Have the right number of abstractions" - The correct number of abstractions is the minimum possible. Every abstraction should be a solution to a problem. You have to define the problem that the abstraction was meant to solve, and then ask yourself if there is a simpler way to solve that problem.
> [12] Be conservative on APIs and liberal with implementations.
> [18] Maintain multiple implementations in test for APIs; compare results between them. The cost is worth it (it will help with correctness, and also prevent leakage of implementation detail).
The word implementation is thrown out a lot. What does implementation mean in these quotes? The databases used (ie. dependency injection), the endpoints (ie. /user, /user/123), or something else?
I'm trying to visualize this in my head and am having trouble.
A large API likely has complexity, security implications, messy transaction boundaries, etc... keep it small and simple, think of UX of an API, make it intuitive and capable.
That's the conservative bit... now the liberal bit.
Make clients, make one in Python and look at date parsing, make one in Go and consider struct re-use and how to deal with sets/arrays of things, make one in JavaScript and consider how easy it it to re-use parts of the code. If you have an endpoint that returns a list of things... is it supposed to be paginated over? How easy is that? Is it an infinite list? How easy is that? What would a client want to achieve? Is it possible?
When you produce prototype applications in a variety of languages the majority of the small inconsistencies and issues are super obvious.
And remember, once the API is out there it's hard to change it and you can't remove things easily... so discovering all of this early helps you get it more right than wrong the first time round.
Implementation is things like, as you noted, DB used, programming language, and so on.
By "Be conservative on APIs" they mean be careful what you promise to your clients as communicated and documented through APIs. Once promised it's almost impossible to go back; you will lose your clients' trust and piss them off. In case of Stripe they are committing to support idempotent requests[2] which is sort of a big deal. And I don't expect them to drop that support in any of their future APIs because I imagine hundreds of millions of $$ business would be relying on that idempotency support.
We have a "StorageEngine" interface that wraps around RocksDB, i.e. "RocksDbStorageEngine" to provide key-value interfaces; however we are worried that someone would leak the implementation detail (RocksDB is so powerful) from the "StorageEngine" interface.
To solve the problem, we write another fully in memory storage engine implementation called "MemoryStorageEngine", which acts as a specification of the "StorageEngine" interface.
We run all storage engine tests against both implementations (rocksdb/memory), except for the durability part; we also configured the system to run over both storage engines.
By doing this:
- we can use the memory based one to enforce the behavior RocksDB based one by running unit tests against both implementations.
- no one can easily leak some implementation details of RocksDB from the StorageEngine interface; to do that, you need to first add those advanced feature into memory based StorageEngine as well!
This is just one example, and we have a list of such abstractions, such as MetadataStore, Logs etc. We created multiple implementation of each of these interfaces and ensure the system could run on any combination of those implementations.
1. We don't want to delete old data so we just use a "deleted" column and do that instead. Now every single time you have to filter out those results. 2. You just made this thing a relationship now every time you have to join.
In the end, if you can keep it simple even if you don't model the data "perfectly". But of course sometimes the complexity is really necessary. To know when is being a good Backend / Db Engineer.
Whether or not these were good decisions depends on the use cases of course, but papering over db implementation details like this with a view is pretty straightforward, no?
Don't screw up the physical tables just because you don't like the knowledge that they contain in certain contexts. Facts are facts. If you don't like what you are seeing, change your query or filter the result set. If a customer record needs to be retained after deletion per biz requirements, then an IsDeleted fact is 100% the right path.
Both of your points enumerated above are trivially solved with views.
Curious how people interpret this? Is it:
* Attempt to keep our API small while also... * thinking widely about the way it's used?
Or how does this work in practice?
Many times, this shifts burden to users. It's always more convenient to have one API which does exactly what you need.
But the idea here is to keep the API simple and stable in a phase where instability is already high.
Perhaps, after it gets mature, it'll make sense to incorporate common use cases into the API structure for convenience.
Someone correct me if I'm wrong, but if my memory is correct, I was under the impression that a pull request was a term coined in the early Github days specific to their PR feature.
Guessing by the context, I'm pretty sure it means "individual contributor", or "person that is not a manager".
I've only ever seen it to mean "individual contributor", but then this would make it odd/bad advice. There should only be 1 "customer". Sure, that customer may have a whole team that actually needs the product, and may provide them for clarification or what not, but to avoid unrecorded/incorrect handshake agreements, specs, or pet requirements that have nothing to do with the initial product ordered there should only be 1 POC for the product unless otherwise specified.
Put differently, interfacing directly with customer ICs is good and necessary, so long as it's done with the go-ahead and oversight of each POC.
I agree that an individual user shouldn't be driving roadmap stuff directly, but getting their perspective, pet peeves, war stories etc. is still valuable advisory input to the overall process.
A lot of time in projects like these you end up getting shunted to tech leads, managers, etc, whose job it is to "take the coordination work" so that people writing code can "focus on implementation". The point of this advice is very specifically to bypass the coordinators and go directly to the people using your project in their code to make sure you understand their concerns, and don't play political telephone.
https://about.gitlab.com/handbook/total-rewards/compensation...
So yeah, it’s a term of industry that often shows up in tech companies because of the usefulness of that distinction in those environments.
Individual Contributor means you don't manage other people, but it doesn't imply you are a team member. It helps to think of it as a role where at some point one is senior/skilled/specialized enough to contribute as a team of 1, working with one or more teams at a time.
(Vouched for this flagged comment after seeing the other comment)
Its not though. Outside of hacker news I have literally never heard this term before, nor does it make any sense. If I'd start talking about "Individual Contributors" to my colleagues they would all be confused wth I'm talking about and wondering what's wrong with me. Who even coined this stupid term, and why do we pretend we need a new name for "engineers"?
The original author saved maybe couple of seconds to write "Individual Contributor" on the first occasion of the phrase, but made many people here spent many seconds trying to figure out (probably by searching the internet) on what that acronym means.
This is especially pronounced in some companies where the acronyms might be understandable just to a few members of some team, and when when this is communicated to the wider company, it can cause even a greater confusion (as you cannot search for it online) and waste of time for such a small "irrelevant" thing.
So in other words, it can be compressed a bit in: "why should I waste a little bit of my time, when I can waste everyone else's time". Or simply: it is disrespecting your audience.
The original article beside that is quite interesting though.