A database for 2022
tailscale.com
tailscale.com
Consider whether the secret sauce here might not be au courant database choices, but rather something much harder for random teams to pick up and run with: just straight up good design and programming.
This is a thing that happens in other fields, too; some of the hardest dishes to cook seem extremely simple and have few ingredients. They're hard because there's no place to hide: your technique has to be flawless or they don't work. A famous entrance exam for new cooks at fine dining restaurants is "make an omelet". Tailscale is, thankfully (and, to me, irritatingly), more like an omelet than a truffled hash.
imo they’re not solving a notoriously hard problem, they’re realizing that they have the tiniest version of this problem (and that will most likely continue to be true). when that’s the case and you want to prioritize stuff that moves the needle like dev velocity, why do anything less simple? getting hung up on building scalable solutions far beyond what you need is somewhere between a fool and a big co employees game. far too often people introduce complexity way before it’s needed, and that almost always slows down your devs ability to actually fix bugs and make the product more reliable.
Which is just a piece of cooked fish, rice, and miso soup. The critic is first almost insulted by such a simple meal. But then realizes the fish, rice and soup are prepared to perfection, and is amazed at the chef’s skill.
Similarly, I’ve also heard (probably from a show like Top Chef, so YMMV) things like a salad or soup are also quick ways to differentiate chefs. In this case, it’s less about technique and more about identifying and mixing flavors.
It works because there is no place to hide. You either have good flavor or not.
To the forum's credit, the top comment (by u/judofyr) the last time tailscale blogged about their unusal db setup, got it exactly right: https://news.ycombinator.com/item?id=25768042
> Interesting choice of technology, but [tailscale] didn't completely convince me to why this is better than just using SQLite or PostgreSQL with a lagging replica.
> In particular [tailscale has] designed a very complicated system: Operationally you need an etcd cluster and a tailetc cluster. Code-wise you now have to maintain your own transaction-aware caching layer on top of etcd. That's quite a brave task considering how many databases fail at Jepsen.
> Considering that high-availability was not a requirement and that the main problem with the previous solution was performance ("writes went from nearly a second (sometimes worse!) to milliseconds") it looks like a simple server with SQLite + some indexes could have gotten [tailscale] quite far.
I think the underlying actual question for Dropbox to ponder (but perhaps not answer publicly) was "what is your moat?" without the business development language and at least for me that remains an open question.
I remember I was in college when Dropbox started and I thought Dropbox was silly to offer more storage for inviting more users. It was easy to approach people and ask them to try something in college. I think I almost maxed out my invites. I remember feeling I was taking Dropbox to the cleaners...
One activity question for HN: why do you think Dropbox continues to exist today while copycats like Barracuda Copy have failed?
FWIW, i've been using Dropbox since practically the day they went online, and have been a subscriber for all but the first few months of that time. They've never once failed me, never been offline when i needed them, never lost anything, they created a Linux sync client out of the box (whereas Google first claimed the would make GDrive for Linux but never did), and it's easy to manage via multiple interfaces (mainly their own web UI and https://rclone.org, which can proxy your dropbox behind an add-hoc FTP server running in your LAN).
They're extremely pricey compared to the same storage in GDrive, but Dropbox has never let me down so i'm in no hurry to drop them.
"...Half-a-billion people stored files on Dropbox. Well, sort of. Really, the files were in Amazon’s cloud. .." [1]
[1] "The Epic Story of Dropbox's Exodus From the Amazon Cloud Empire"
https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...
The challenging part is to provide the back-end. Get massive storage, with proper checksum integrity, at a cost effective value and with flawless security.
A single command (and minor variations around it):
aws s3 sync <name_source_dir> s3://<name_target_bucket>
get's you all that. It can be easily mapped to an intuitive user interface by a software development boutique in weeks.
And because that is the difficult part, it seems for the first 8 years of their history that was solved for them by AWS, according to their own reported history.
Sharing a folder or file from Dropbox is no more intuitive than using the S3 console. Looking at the very colorful history of security issues around Dropbox [1] it's certainly not safer.
So why would a user pay more for storage costs than with a cloud provider, with notoriously less security? Does a tiny layer on top one or more cloud providers justify the add-on value for the user or the current valuation of 9 Billion dollars?
[1] "Criticism of Dropbox": https://en.wikipedia.org/wiki/Criticism_of_Dropbox
side note: I'm filled with regret for every document that I authored on Dropbox Paper. It didn't last for long, but there was a time that it was the official wiki for a company I worked at. Its not impossible to export, but never with full-fidelity.
I've not really used the S3 console but, at least on macOS, sharing a file from Dropbox is as simple as "right-click on the file in Finder, Share..." then enter an email address or create a share link from the popup. Or I can right-click and "Copy Dropbox Link" if the direct link is good enough.
Is the S3 console simpler than a right-click?
aws s3 presign s3://awsexamplebucket/test2.txt --expires-in 300
How is this easier than "right-click, pick a menu option"?
(If I want an expiring Dropbox link, it's there under 'link settings' in the pop-up. Admittedly only has date granularity but still there as an option.)
Not that I like Dropbox too much. I'm going to switch to my self hosted solution. But it's certainly not as easy as using Dropbox, even as someone who uses AWS since 2012 frequently.
> Wouldn't also be HN without the misinterpretation of BrandonM's response. Here is dang's comment about that topic/meme: https://news.ycombinator.com/item?id=29178442
Its different reading it at the time because it factors in the context of the community at the time as well as the lived experience of what it was like at the time to be a member or participant in given discussions. With that contrast, you can then get an idea of the intent of those comments beyond just the words, as they are often (and in this case) reinterpreted through the lens of current day contexts.
Example: ask what people think of the hit TV show The Apprentice today in the present, then hop in a delorean and ask someone before the trump presidency or candidacy what they think of the show.
You can call me anytime <3
This article could have easily been "why we were down for a day while some auto-vacuum setting was broken." Then everyone would be like "why didn't you just scp a json file between machines" ;)
Say "switch databases on a successful project" and I bet people's minds go there. That is indeed expensive and painful, but if you have a small team, it's just being agile.
There is also the "punching up" / sour grapes aspect that can't be ignored. "I would have picked the right database from the start, but nobody's funding my crazy idea." Life just isn't fair, and HN is where you can vent, I guess.
The kinds of projects that I more generally tend to see make these migrations successfully are the ones that design with in-memory data structures first and then sort out how best to persist that kind of thing to disk after their domain model has solidified a little bit.
Tailscale smells a lot like that to me.
But as I’ve been using only statically typed languages the last few years (on the backend), I’ve moved to designing the domains in the code, like you said.
When Tailscale migrates databases, it is a few engineers working for a few weeks solving problems that already exist.
When a company with 50k employees does it, it takes two years and only makes things worse.
My go to language of choice is PHP/MySQL. I've never not been able to scale a project. At one point while working at Comcast I prototyped a system for every error code from every cable set top box (150 million) in the country on a MySQL server instance running on a MacBook Pro pulling data from Splunk in real time. All so I could generate some png charts and embed them in Slack.
I know there are limits to MySQL but so far in my career I haven't had any projects hit those limits because hardware has been getting better faster.
That Comcast prototype sounds amazing. PHP and MySQL can really bring you very far, i use it for nearly everything and it is so incredible easy to develop and “deploy” new versions. I use propel as ORM and it is very close to the DB and I can change schemes fast and just let the autoupdate handle the rest.
A k8s cluster for something like that wouldn't cost anywhere that amount, plus you would get the advantage of being able to deploy your app in multiple availability zones. If you don't need k8s features, there are definitely simpler options that will give you redundancy and won't break the bank.
I have gone past mysql limits a few times. Will you go beyond its capability is a simple measure. Can your hardware write fast enough to keep up? If your hardware maxes out writing to disk 200 megs per second, welp mysql maxes out at 200 megs per second. If you need 400 megs per second of writing, well you need two instances that can write independently. That is when you are fucked. At that point you get into sharding* and doing whacky things mysql isn't meant todo. Reading things from disk, mysql's replication scales just fine.
The replies are focused on MySQL (and to a lesser extent, PHP). None of that matters.
The hero here is the logging system serving as seekable stream of collected events. (I’m trying hard here to not say K-fka).
By consuming events from that source and being able to perform time-based seeking, you’ve offloaded (or dare I say, eliminated) your scaling pressures.
Even if you commit the “sins” of no indexes on heavy queries, using MyISAM, or building a giant, memory-intensive associative array… none of it matters.
You can choose to wipe your local dataset, start again in a different language, switch to another data store model, and none of your downstream consumers will know nor care (unless you blog about it).
*This* type of design, that prioritizes consumer-facing resiliency and continuity, is what I look for in systems that scale.
Later on i met the roomate of the author of memcache. Few hundred lines of C. Completely changed the game how caching systems worked. Years later a super senior FB engineer explained to me over a beer - "we basically use mysql as a file store, everything else is in memcache." Good ideas are stupid simple.
You mean Ron Jeffries?
A discussion that keeps recurring on HN is how you can get a lot done with simple, boring solutions. This is a perfect example of that in practice. What’s the simplest thing that’ll work? Do that, monitor the solution, and when you start hitting a limitation reevaluate what the next appropriate solution is.
There are plenty of comments here deriding this as insanity when we have established best practices, but “best practices for me are not necessarily best practices thee”.
People aren’t perfect, not every engineer comes fully-formed grokking the industry’s best solutions and their trade offs.
This is what learning looks like. This is what an evolving system looks like. And this is how valuable software gets built every day.
Who gives a fuck about scalability on a poc? “Let’s just put in a json file”. I Love it. Personally I start at the “use SQLite” step and go from there because SQLite is such amazing software. Thanks to this post I’m excited to try out Lifestream.
You might rightly point out that Tailscale isn’t a side-project/poc anymore. But that’s exactly why they’re changing to SQLite and writing this post; this is their team learning to deal with larger problems and changing needs, and they’re sharing that journey with us.
As long as they’re not _ignoring_ serious risks let them experiment. But try not to get too upset if they’re making different risk/effort trade-offs than you would. Even if it’s “objectively” the “wrong” thing.
My rubric for technical sanity is:
- are you taking (and testing) backups?
- is your infra/data secure?
- are you monitoring what’s happening?
- can you recover from catastrophic failure in a acceptable time frame?
- are you meeting your legal obligations?
Cover those needs and you can be forgiven a lot suboptimal/experimental implementation details.
The thing is, David Crawshaw is being extremely honest and humble here in admitting their shortcomings and how they're learning from it... but an entire troupe of Silicon Valley engs seem to have lost sleep over it, for reasons beyond me.
There is negativity on this thread, but it strikes me as incredibly, even embarrassingly, ill-informed.
From what I understand RDS works great and you’re happy to pay what feels like the tiny cost…right up until it doesn’t anymore.
You’re paying Amazon a premium for them to manage the tedious stuff. But sometimes dealing with the tedious is what teaches you how the stack works and makes you jmuch more prepared to solve those more complex problems down the road.
The uncomfortable fact is businesses finding themselves in situations where they experience vendor-lock-in just because they’ve let their capability wither away.
Clearly David’s point about avoiding MySQL and PostgreSQL due to fears of “vendor lock-in” resonates with me, even if I personally think that’s not quite where the threat is.
…for Postgres anyway. I’ve heard enough stories that I never want to touch an oracle product in my career.
I'm curious how these tools will mature, it seems like a good match for microservices.
It was used by https://github.com/blevesearch/bleve, https://github.com/etcd-io/etcd, and number of other projects.
These days, https://github.com/dgraph-io/badger is often favored because of it's improved write throughput.
https://news.ycombinator.com/item?id=30015913
(This comment is not to suggest that the Litestream project is of poor quality or anything like that.)
Postgres wire compatible SQLite proxy - https://news.ycombinator.com/item?id=30875837
Sometimes boring is the right choice. PostgreSQL has worked for decades now, and seems to have regained much of the performance that MySQL once boasted.
If you do this for the money, investing in tried and true (but boring) software should be the default solution.
I watched with dread how the MongoDB fiasco played out a decade ago. Meanwhile, I kept using PostgreSQL and had a good ride.
Things are complex, especially in data, I think it’s important to evaluate use cases rather than going with generic assumptions like “use boring tech”.
When you want to deviate from the boring software and technology, there must be a very sound 'business case' for it.
If you are building a planetary scale business, OK, maybe that warrants something more fancy.
But how many companies really need that? I'm not limiting myself to the HN unicorns, but looking at the whole market.
> If you do this for the money, investing in tried and true (but boring) software should be the default solution.
Meaning if you are just playing around, then by all means evaluate new and shiny stuff.
IMHO MySQL becoming boring (fast, pretty reliable, non-compliant quirks being ironed out, decent backup tool availability, credible admin tools, JSON and CTE support) is one of the most important long term trends on the internet.
Along with the parallel evolution of PHP.
All the PHP+MySQL I've worked with has been horribly bespoke and brittle.
Want to significantly change a URL? Need to refactor every path in the .PHP file, because it's importing something from ../.. Or get in the spaghetti business with path aliasing on the webserver.
Want to refactor the database schema? Difficulty: Impossible, because raw SQL strings are scattered everywhere, including on other systems.
Want to add/modify a new internal CRUD form? Gonna be several hours of work to RE what the last guy did to keep things consistent, while tiptoeing into the SQL to not break anything. With a proper framework, this is a 5 minute task and a few LOC into the "admin interface".
It works, and you can do a lot with just a few php files and a basic Linux system, but there are downsides.
Laravel is, IMO, among the best web frameworks available full stop. Certainly one of the most maturely documented.
It addresses all the points you make.
Eventually some blade template error resulted in an impossible stack trace due to the insane amount of magic behind the scenes in Laravel, and while debugging that I decided to stop relying on such black magic hack of a framework. Around the same time Go gained in popularity and I have been working with Go since and are very happy with it.
This was Laravel 5 days and I'm sure much has changed.
I completely adored the laracasts made by Jeffrey Way. I wish every framework had a guy like that do produce learning materials.
Frameworks solves a general problem, you on the other hand has a specific problem. To be able to accommodate your specific problem and everyone else specific problems frameworks has tendency to heavily rely on magic.
* magic classes - using PHP magic methods for everything, the actual class is usually empty and things happening elsewhere.
* magic files - put a file with the correct name in folder somewhere and things happen
* magic configuration - put a hard coded string in some config file and you change the entire behaviour of the app.
* magic layers - if you follow the execution path with the debugger you spend the majority of time circling around in different layers and very little time in your own code.
All of these things are considered bad practice when doing it in your own code, but for some bizarre reason it is considered good practice in frameworks.
I think that’s because that kind of code has a much higher maintenance burden. It’s much more difficult to get up to speed on what’s going on in complex magical code. But you can probably trust a framework that’s achieved critical mass to make reasonable decisions and stay alive for a while.
And if I'm gonna deploy a new framework greenfield, I'll just go with Django since Python has more mindshare in my industry.
I have created many on-the-fly mini-PHP frameworks to use within existing legacy code bases to avoid all of this.
Easiest way to avoid manually requiring files is to use classes with the auto loader. Done.
The second easiest solution is to have different constants that has the full path to a directory and then just you use the correct constant when doing a require. No relative paths required.
You can even have one require constant per file if that is your game.
Raw sql is fine, just don’t scatter it in every file, instead put sql in specific repositories, e.g ArticleRepository, and do all article queries in there. Now it is easy.
An ORM actually creates more problems when refactoring because entities are spread out everywhere.
If you use something like Phpstorm IDE it can help refactor raw sql too.
Problem is that many developers doesn’t think this thru and even worse they accept the situation and don’t do anything about even if they can. Instead they go to forums and complain. I’m sorry but I have zero tolerance for that.
Laravel has an excellent ORM, a great query builder (I write almost no SQL), an effective database migrations system, a solid class hierarchy (no include statements anywhere), a really useful job queuing system, and it is built around Composer (one of the best package managers anywhere).
Lighthouse is really a first-class, schema-first GraphQL binding with a very logical code interface.
I don't see any code maintenance problems in Laravel that can't be mitigated by the same discipline you need anywhere else.
If you have a solid team and some people with a good background in the framework and PHP that can mentor effectively, then it probably won’t be bad. It still would not be my first choice though.
Could you elaborate on this? MongoDB as a company is worth $30B, so it looks like they did at least some things right.
There was a Slack competitor using MongoDB that was snappy and js hipster trendy that ultimately failed because of unreliable database IIRC.
It also works well for early stage Consumer Internet startups that haven't yet achieved huge growth but care a lot about developer productivity while churning features at high velocity.
But it gets blamed for reliability issues suffered by highly successful Consumer Internet companies that have achieved scale and have DAU, MAU in > 10M and have lot more than 4TB of data in a single cluster.
But that's basically a good problem to have at that stage. Usually such a company would have many more problems – monolithic application with monolithic database with huge unmanageable schema and indexes gone wild etc. Usual solution at that stage would be some sort of rearchitecture towards Microservices with multiple specialized databases for different use-cases – usually cloud hosted managed databases with horizontal scalability dedicated for online user-path workloads and separate OLAP tech stack for offline ETL/analytics workloads.
The relational model is... JUST... SO... GOOD. And, it is a shame that most of the relational systems are so complicated.
A document within Adama (https://www.adama-platform.com/) is basically a giant JSON file held within memory with clients connected via a WebSocket. I'm basically building my own indexing since I want the indexing to be reactive, and I've got exceptionally fun optimization problems.
Litestream, in my opinion, is a great way to get started. Actually, it's beyond fantastic if you maintain the 1 server to 1 database because migrations become so easy. So, I applaud the team for taking this simple approach.
(we don't use Litestream right now; we're pushing the complexity up a layer in our design instead)
A lot of people on this thread are looking down their noses at sqlite, but I think they're kind of beclowning themselves; the unreasonable effectiveness of sqlite has been a meme in infra dev for a couple years now. It's not a new idea. Lots of people are doing stuff like this.
The funniest bit on this thread is the person saying they should use RDS, as if their infrastructure was just a big Rails app.
(I think the reason they gave was vendor lock-in, but apart from that, I didn’t understand why it wouldn’t be adequate)
We use Postgres! It's the backing state for our API, and an important source of truth in our architecture. Postgres is a great way of serving a GraphQL API. It is not necessarily a good way to back an infrastructure service.
This isn't an app server; it's an infrastructure component, running (I don't know about Tailscale here, but we use SQLite in similar uses cases) on potentially hundreds or thousands of machines.
For example, if you want to index a "foreign key" id stored in each "record" in a JSON array of objects, you build a hash table from the FK id values to the JSON array indices of the objects that have that id. It can be as stupid simple as an `fk_index = defaultdict(set)` somewhere in your program, to use a Pythonism.
Now when someone wants JSON objects in that array matching a given FK id, they can just O(1) look in the index to know the position of records that match. Much better than an O(N) scan of every item in the array.
Of course you have to to maintain the index as writes to the JSON happen, but that's not bad once you understand how things work. No real secret sauce.
So, I mirror this thinking and having be just an object with the keys being the primary key. Then, I simply index all the children by their fields based on insights from the developer via the index keyword.
So, if you have
record R { public int id; client int owner; int age; index age; } table<R> rows;
then queries for age can be accelerated by the table.
like "iterate rows where age==42" will basically hone in on the bucket of age==42. I currently only index clients by hash and integers.
The critical aspect which makes this work is that I monitor all mutations. When a child object has a field mutated, then it is removed from all indices and placed into an unknown index. Any queries will also consider it as the purpose of queries to simply narrow the field. Once data changes are persisted, the index is updated and items are moved out of the unknown bucket. This works fairly well because the indices are primarily used during the privacy check phase.
I seem to recall Richard Hipp on the "changelog" podcast mentioning it, but I don't remember what he said.
2 episodes here: https://changelog.com/person/drh/podcasts#feed
I think it was very neutral, something to the effect of "there are multiple solutions".
But I know essentially nothing about sqlite internals so I can't judge, or maybe that's why I didn't understand what he said.
I guess I'm wondering if it's a recommended/supported mode of operation in sqlite. What are the failure cases? How much data can you lose?
Litestream author here. Dr Hipp and his team reached out when I first released Litestream and we had a video call together. They were fantastic. Really friendly and down to earth. I explained how I put together Litestream and we went back and forth on different approaches. They were really helpful with understanding some of the shared memory stuff that's what makes the new read replication work. They ended the call by asking me if there's anything they could help with or anything I needed.
> I guess I'm wondering if it's a recommended/supported mode of operation in sqlite.
I don't think the SQLite team endorses any tooling outside of what they build AFAIK. So in that sense, no, it's not officially supported. However, the API for maintaining checkpointing is publicly available and there are docs for controlling it from outside processes (which is what Litestream does).
> What are the failure cases? How much data can you lose?
Litestream is designed so that it keeps retrying in the event that you can't connect to a replication destination. S3 is pretty reliable so it may be a network outage that could cut you off. You can also enable Prometheus metrics to be reported out of Litestream if you want to add monitoring and alerting.
As far as data loss, by default it's setup to bundle database changes together every 1 second, compress them, and upload them to somewhere like S3. So your window for data loss is 1 second unless S3 goes down.
You can also run regular backups with the SQLite CLI and upload those as a fallback. That's what I typically do since it's really cheap and easy to setup an hourly cronjob and I'm overly paranoid. :)
/dream
FoundationDB is similar but you have to do so much yourself. It's more or less what I'm describing for a key value store though. Not sure why it's not more popular.
It is weird that this DB has a good reputation which doesn't match its popularity.
The index backing the unique constraint on the URL table for my search engine is around 23 Gb. The entire server has 128 Gb of RAM, for comparison.
I arrived to a design where SQLite databases would be synchronized by Kafka. Kafka is really robust, and has a friendly semantics when configured to be in-order delivery.
The catch is, you don't issue writes to SQLite anymore. You write Kafka messages, and have to prepare to resolve concurrency problems at read+write-to-db time. For example, user registration is not an atomic action anymore - you write a registration attempt message, and then it might race with a concurrent registration attempt from an other node (say due to retries and random load balancing).
You resolve the race in message-reception time (doable due to stable order on multiple reader nodes). I expect this would work nice, but needs you to rethink all actions that used to be synchronous db writes.
These days I wince when I hear of overcomplicated solutions for an MVP that "will scale" at a nebulous point in the future, while we're going to be paying the complexity penalty every day for a benefit that may never come.
Using openssl with a wrapper service? Doesn’t seem particularly dubious to me.
Coincidentally, one of tailscale's founders already built this: https://github.com/sshuttle/sshuttle
Don't know why anyone would use SQLite in production except for as some KV store embedded in an app.
The vibe I get from this thread is that a lot of people have only experienced SQLite from the vantage point of something like Rails running in a "test" environment, and are used to explaining to people on some PHPbb somewhere that you can't really use SQLite to back a multiuser SAAS app. That's as may be, but when you're building infra components, you control who the "users" are; not everything is a web server taking requests from all comers.
Do you understand what kind of application Tailscale is? For all intents and purposes it's exactly what you described above.
Why I Built Litestream - https://news.ycombinator.com/item?id=26103776 - Feb 2021 (176 comments)
- one leader that can write to the db
- a ton of other hosts that can read from the "streamed" db and want to do so often/fast
- hosts send their writes (if they have any?) to the leader via some api call (?)
Is that right? If so, the solution makes a ton of sense. "Just use pg/MySQL" in that case would mean hundreds (thousands?) of read replicas, which doesn't sound fun, or every read over the network.
But tailscale isn't some random group of engs. They've probably got the chops to pull off literally anything they want to. I mean TFA casually mentions online cross-database transfers, multiple zero-downtime schema migrations, inspecting litestream's replication code for feasibility, deftly modifying sqlite WAL checkpoints... all in one breath.
It seems to me that tailscale engs want to avoid DBA work but also not use managed offerings, and so, they're comfortable paying the costs they have to (such as multiple migrations).
> ...rather than improving the product?
Well, you'd guess they want to be able to continually improve their already credible product too. When TFA points out that zero vendor lock-in and hassle-free, local end-to-end tests are non-negotiable, I think it is for this reason.
----
> if they're this talented, is their time really best spent on...
From: https://tailscale.com/blog/go-linker/
"People are often surprised and sometimes horrified when they learn that Tailscale maintains its own fork of the Go toolchain. Tailscale is a small startup. Isn't that a horrible distraction, a flagrant burning of innovation tokens?"
"Maybe. But the thing is, you write code with the engineers you have."
"We had a problem: We kept crashing on iOS, and in addition to being awful, it was preventing us from adding features."
"Another team might have decided to cut even more features on iOS to try to achieve stability, or limited in some way the size of the tailnet that iOS could interact with."
"Another team might have radically redesigned the data structures to squeeze every last drop out of them."
"Another team might have rewritten the entire thing in Rust or C."
"Another team might have decided to accept the crashes and attempted to mitigate the pain by making re-establishment of connections faster."
"Another team might have decided to just live with it and put their focus elsewhere."
"The Tailscale team has Go expertise, spanning the standard library to the toolchain to the runtime to the ecosystem. It’s an asset, and it would be foolish not to use it when the occasion arises. And the fun thing about working on low level, performance-sensitive code is that that occasion arises with surprising frequency."
"Blog posts about how people solve their problems are fun and interesting, but they must always be taken with a healthy dose of context. There may be no other startups in existence for which working on the Go linker would be a sensible choice, but it was for us."
if zero vendor lock-in and hassle-free, local end-to-end tests are non-negotiable, why are they using s3? migrating to another s3 compatible backend would be similar in effort to migrating from aurora mysql or postgres to another managed mysql or postgres service or to self-hosted mysql or postgres
You may be right. I have no experience migrating litestream but from the docs (https://litestream.io/guides/) it is literally cp'ing files from S3 to wherever and exec'ing one of these one-liners (of course, the devil is in the details):
litestream restore -o my.db s3://BUCKETNAME/PATHNAME
litestream restore -o my.db abs://STORAGEACCOUNT@CONTAINERNAME/PATH
litestream restore -o my.db gcs://BUCKET/PATH
litestream restore -o my.db s3://SPACENAME.nyc3.digitaloceanspaces.com/db
litestream restore -o my.db s3://BUCKETNAME.us-east-1.linodeobjects.com/db
litestream restore -o my.db sftp://USER:PASSWORD@HOST:PORT/PATHIt's a super confusing argument regardless, because the industry is lousy with "S3-compatible backends".
I just don't believe the tale of "such skilled engineering teams" which don't show that in their products but blogposts.
RDS has some significant downsides which I would personally consider no go if I were in a position to evaluate it. The one which stands out as particularly painful from my past experience is… it’s excruciatingly slow to provision or make configuration changes. Like lose whole days of work to a few iterations of trial and error slow. Combined with AWS’ sprawling and inscrutable set of authorization and configuration options, the weird idiosyncrasies between most of their offerings, and the absolutely opaque naming applied to most of those offerings… trying to use RDS effectively as a managed database service felt more to me like becoming a full time ops professional.
Edit: I just found this tips and gotcha page: https://litestream.io/tips/
If you're looking to have multiple primaries then you can try using the SQLite session extension[1] and copying patch files. That's quite a bit more complicated though.
Replication of data has always been a concern for us, with the current mitigation being periodic VM snapshots. In order to engage larger customers, we would need to tighten this up.
Our next gen data storage technology leverages some dark magic from fintech. Append-only log, single writer, synchronous replication, etc. Most recent benchmarks with replication to 1 sync witness on same lan are in excess of 200k business objects created or updated per second.
What about managed mysql/posgresql? no lock-in and installing them locally is trivial.
AWS does offer one (very reliable) implementation but they’re very definitely not a monopoly.
That is my conclusion, too.
There's also a strong correlation between people who look at things from weird angles and also build good products. Why are you surprised the people who invented an entirely new way of doing VPNs also don't cargo cult database storages?
maybe next they will stop cargo culting operating systems and switch to SerenityOS?
Tailscale’s coordination server, our “control plane”, has become known as CONTROL. It’s currently a single Go process on a single VM.
source: https://tailscale.com/blog/an-unlikely-database-migration/
This is what a really smart and future-proof solution looks like.
I'd put money on this not being the last database change too! But not for the same reasons you would.
I'm not joining the "just use RDS" crowd, but it looks to me like using a managed DB service like RDS would have been a reasonable decision for this application.
[1]: https://tailscale.com/blog/an-unlikely-database-migration/
You could absolutely run a web site using Brainfuck without any failures or any customers realising it. Doesn't mean it's not a questionable decision!
I did briefly have logical replication working, and so had my near zero downtime solution in hand. But the replication broke after a few days because I delayed cutting over, and then I couldn't get it going again. I couldn't find out how to debug this either.
The lesson I've learned here is that I'll be avoiding managed DB products whenever I can. It takes away control that turns out I sometimes really need.
On another note I’m using Tailscale and it’s a wonderful service and I get it for free.
Some comments say that it's because they're complicated but what exactly is complicated about them?
Can someone with more experience explain to me what would have happened if, for example, they used PostgreSQL from the beginning?
P.S Apologies if this is a stupid question.
Edit: Looks like I hurt someone's feelings in this comment.
> Tailscale’s coordination server, our “control plane”, has become known as CONTROL. It’s currently a single Go process on a single VM.
if they had used postgres/mysql or some other external RDBMS then that would have meant managing a whole other set of dependencies and doing at least an IPC/Pipe roundtrip for every single query instead of being able to use an embedded database like custom-built JSON db or SQLite which is only going to need in-process syscalls for reading/writing to disk/cache.
so, this allows them to scale their queries vertically orders of magnitude more than an external db (SQLite can do an insane amount of simple read queries per second) as well as making their whole dev/test/release cycle simpler and easier to manage.
that's my understanding anyway - hopefully it makes some sense.
> Unfortunately, if we want to move to running more than one CONTROL process simultaneously for high-availability and better release management, that means we no longer have exactly one process managing the data
Do you guys still only have 1 CONTROL process?
> We’re going to put everything in a single file on disk again.
For most small scale projects I'm just reading and writing lines to a file on disk. Sure, I could over-engineer some database, or I could get a minimal viable project off of the ground and worry about scaling later. Most projects run into issues long before scalability.
I found myself recently wanting a lightweight key-value store for random structures. For this purpose, I created a toy single-file key-value store [1] in less than 256 lines of C. It's not quite ready for prime time, but it's incredibly easy to use.
anyone has an example of how to do this?
Hasn't AWS's SQL db-as-a-services had this for years now?
May be Oracle remains an option. ;)
Take a look at https://litestream.io/guides/systemd/ and compare that with CockroachDB's.
In all fairness, litestream is no CockroachDB/PlanetScale/YugaByte; but why pay the ops overhead (and your lawyers to make sense of their "OSS" licenses) when litestream would do just as nicely.
TFA goes (emphasis mine):
Litestream..grabs a lock so that no other process can checkpoint. It then watches the WAL file and streams the appended blocks up to S3, periodically checkpointing the database for you when it has the necessary segments uploaded. ___This gives you near real time backups and lets you replay your database from S3 trivially..It's a great hack, and a small enough program that I could read my way through it entirely before committing to it.___
[0] - https://github.com/cockroachdb/cockroach/blob/2c4e2c6/LICENS...
[1] - https://github.com/cockroachdb/cockroach/blob/2c4e2c6/pkg/kv...
TL;DR is the source is available but you can't use it in production without paying a fee. However versions older than 4 years become truly open source, so you can use old versions for free.
Interesting compromise. Might be better than "open core".
> The Licensor hereby grants you the right to copy, modify, create derivative works, redistribute, and make non-production use of the Licensed Work. The Licensor may make an Additional Use Grant, above, permitting limited production use.
Maybe specific software modifies that as you say, but the base BSL doesn't. I agree it's super vaguely worded ("production" isn't defined at all). Also it turns out MariaDB itself isn't BSL licensed - it's GPL, just some extra projects around it.
See the text of the Additional Use Grant for CockroachDB [1]:
> Additional Use Grant: You may make use of the Licensed Work, provided that you may not use the Licensed Work for a Database Service.
> A “Database Service” is a commercial offering that allows third parties (other than your employees an contractors) to access the functionality of the Licensed Work by creating tables whose schemas are controlled by such third parties.
(disclaimer: I work at Cockroach Labs)
[1] - https://github.com/cockroachdb/cockroach/blob/master/license...
Edit: no it's not..
> Footnote: coworkers point out it’s April Fool’s today and request that I clarify this isn’t a joke. Joke’s on them: every day’s April Fool’s in the Tailscale Database Engineering department.
* Instead of an actual database use a JSON file.
* Write a blog post about how that didn't scale.
* Instead of an actual database hand-roll something else.
* Write a blog post about how that didn't scale.
* Instead of a database with built-in replication which is tailor built for key-value storage use SQLite with a single table containing key-value pairs and some fresh glue to make it replicate.
Are we supposed to expect another blog post somewhere around a year down the road titled "SQLite didn't scale" or "The new library for replication that was released a few months ago contained a bug that took our prod down"?
Is there a reason the replication needs to be built-in?
* SQLite which I mostly see as an embeddable database for applications, used on a server.
* Single table with key-value pairs.
* Having the tool for replication be a quite young project that's in active development.They have a single table with KV pairs right now as an artifact of having used the "database with built-in replication which is tailor built for key-value storage" you complain that they're moving away from. The point of the blog post is that they're doing that in part to move away from the single table of KV pairs.
They're not "sharding".
This is common but incorrect view of SQLite. It's able to handle many server-side workloads, as long as you're aware of its limitations (and the Tailscale team certainly are).
Our database needs are tiny, as explained in the earlier post. So we optimize for things like: "can we run all our tests quickly and easily in many environments without containers and VMs?" All three of our storage schemes have had that property.
We have MySQL and PostgreSQL veterans on the team. We know those options well.
Why?
The official docker postgres package weighs in at 100MB. Assuming you standardized on that, then every environment would end up having a fresh postgres image just waiting to be startup. Meaning, the actual cost is the memory for the server and startup time, not the image itself.
Mount the data on tmpfs and you can startup and setup a db in very little time (I know, because that's what we do).
You might be able to eek out a bit more performance by hand tuning test files... but.. to what end?
> Zero config VPN. Installs on any device in minutes, manages firewall rules for you, and works from anywhere.
I don't know how their product works, but my intuition is that your suggestions aren't feasible.
> MySQL (or PostgreSQL) would come next. I’m not particularly familiar with anything MySQL post 1998, but I’m sure it would work. The HA story for open source databases is somewhat surprising, though: you can either have traditional lagging replicas, or commit to no-primary-replica clusters that have very surprising transaction semantics. I wasn’t excited about trying to design a stable API or good network graph calculations on top of those semantics. CockroachDB looked very promising, and indeed still does! But it’s relatively new for a database and I was a little concerned about getting attached to features in a fresh DBMS that would be hard to migrate away from if we needed to.
The reason they are ruling out products isn't "this won't work where it runs" its "I don't like this". They talk about worrying about network graphs and proceed to use SQLite and copy garbage around.
We're moving towards sqlite for things for the same reason. sqlite is simple to reason about. The distributed state problem not simple, but our distributed state problem is not Tailscale's and theirs isn't yours; every distributed state problem is unhappy in its own way. As they work out the contours of their specific distributed state problem, sqlite isn't the component that's going to break down.
Similar to the current engineering bills they are paying because they switched from a json file -> etcd -> sqlite?
Seems like the cost of switching isn't really an issue for them since they've pulled that lever multiple times now.
At some point, migrations were taking too long, so I just started caching the database schema. I have a go generate thingie that creates a fresh database, applies all the migrations, and dumps the schema to a file that gets checked in with the code. (CI checks that the migrations actually result in the database state that's checked in. Yes, there is something that regexes out the hostname from the sql file, since Postgres dumps that in there by default and that results in spurious diffs.)
I have learned that people often put up with really sub-optimal developer workflows. My test is that you should be able to add "print hello world" to the top of your main function, and see that printed on your screen in less than 5 seconds. (Yes, even if your application is complicated. I have a hacked up copy of K8s that starts that quickly, so that you can write an app that deeply integrates with K8s and not have to reuse a cluster between tests, or pay a long cluster creation cost. All obstacles must be removed!) If it takes longer than that, you are just throwing your developer's salaries into the toilet. (If it takes 6 seconds, you open up HN, and there's an hour gone!)
How's that an important metric to optimize for? Did you ever benchmark the tests with other solutions?
edit: also don't forget these blog posts are content marketing for their product. that's really the point anyway
they're not going to be about shoe-horning a JSON file or a SQLite database file into my enterprise product.
Yes, but ongoing maintenance isn’t free.
If anything, this whole DB funkiness indicates to me just how nice a product you can build despite it.
I struggle with this a lot in my writing too, but I've drawn a conclusion I'm sticking with: it's better to relay what the team is actually doing, "warts" and all. We ranked on HN a few weeks ago with a post about user-mode WireGuard that was basically a litany of engineering mistakes, and it's one of the things I've most enjoyed writing.
Every serious product gets built the way Tailscale is describing. You start some components with the simplest thing that can reasonably work, and see how far you can take them. The JSON file Tailscale used to use was one of the most interesting and valuable things they wrote about.
If you want content about how to built best-practices-compliant hyperscaler-grade systems, there are places to go to get that kind of content. But you've never had a problem with Tailscale that owed to their JSON file backend not scaling --- how I know that is, you've likely never experienced a stability problem with Tailscale at all, because it just doesn't break. That they got it to work, for a long time, with a JSON file backend should make you think harder about how you build things.
Which...worked. For quite a long time. And required no dev effort. Freeing devs up to focus on other things.
They did it as a POC, and it worked, better even than they had anticipated; why over-engineer it up front? Even when it turns out they are gonna need it (i.e., YAGNI no longer applies), they have been able to push off their "solve problems customers don't care about" to instead favor "problems customers do care about". That...seems like sound engineering to me?
When you reach capacity and it hits a screaming halt as the hardware capacity has been reached and performance doesn't collapse its completely gone.
This is not about best use. Hell json file performance may be well within spec of the system requirements, but jumping up and down without perf numbers or specs is typical of sweep it under the carpet for a generation...
It depends on what that tool is doing and if the thing it's doing is a core competency for your company.
It absolutely makes sense to geek out about stuff like this if it's part of your core product. However, if it isn't, then do yourself and your engineering team a favor and pick boring technologies.
Do you know what DB system tailscale would be using if they choose something boring postgres or Cassandra? Postgres or cassandra.
The cost of not doing that is they are on their 3rd iteration of switching db technologies and the development effort to keep rediscovering the same problems.
sqlite is amazingly simple for local dev. Probably even simpler than a json file.
High availability Postgres is complex to operate (you need a whole separate strongly consistent data store to coordinate it). Cassandra is also very complex.
sqlite is, again, simpler. Litestream makes sqlite plausible for production.
You are buying simpler local dev with a more complicated and error prone production.
> High availability Postgres is complex to operate
High available SQLite is complicated. More complicated by the fact that SQLite wasn't designed at all around the notion of being a multi-tenant database system. Instead, you've got to bolt on solutions to fix the fact that you chose SQLite.
Containers make local dev with postgres or other db technologies pretty slick. I have a hard time buying the "oh no, but then we need to run docker" because docker containers have added benefits often simplifying local dev (All your dev tools defined in the docker file. Done, new dev installs docker and moves from there).
> sqlite is, again, simpler. Litestream makes sqlite plausible for production.
SQLite is a fantastic tool, when it's used in the right location. A backend database is not the right location for SQLite. Litestream may make it "plausible" but it certainly doesn't make it a sane choice.
How so? Any evidence for that?
> More complicated by the fact that SQLite wasn't designed at all around the notion of being a multi-tenant database system.
Neither was Tailscale, that's why the engineers decided they were a good fit. But I'm sure you know more about it than they do.
> Containers make local dev with postgres or other db technologies pretty slick.
No they don't. Dockerizing everything for local dev bogs you down to Docker's added latency for every operation, this is not even getting into the latencies added by having to periodically `docker pull`, `build` images, provision the disk space for the images, and maintain all that infrastructure over time. Compared to that, an SQLite database is basically nothing.
> All your dev tools defined in the docker file. Done, new dev installs docker and moves from there
Docker is not a good way to do local dev, overall. The entire point of local dev is speed of iteration. If you need a usable test environment, set one up in UAT.
> A backend database is not the right location for SQLite.
The SQLite team actually disagrees with you on this: https://www.sqlite.org/whentouse.html
> SQLite works great as the database engine for most low to medium traffic websites (which is to say, most websites)....Systems designers report success using SQLite as a data store on server applications running in the datacenter, or in other words, using SQLite as the underlying storage engine for an application-specific database server.
But again, I'm sure you know better than them.
IMO both this and "announcements of new, perfectly-formed features" are at the same level (if not lower, because a new feature is still something new). Both are essentially marketing with no technical content that's really worth your time. Except that it's now the third iteration of "we did an obviously questionable thing and replaced it with another questionable solution".
> it's better to relay what the team is actually doing, "warts" and all. We ranked on HN a few weeks ago with a post about user-mode WireGuard that was basically a litany of engineering mistakes, and it's one of the things I've most enjoyed writing.
I can agree with that, but I wouldn't bunch your writeup into the same group. That's some pretty decent technical content about the problems that were encountered. Meanwhile in these writeups what were the problems? "JSON file database too slow", "Hand-rolled software has to be maintained in-house"?
> Every serious product gets built the way Tailscale is describing. You start some components with the simplest thing that can reasonably work, and see how far you can take them. The JSON file Tailscale used to use was one of the most interesting and valuable things they wrote about.
Yet they keep changing it, migrating, writing tests for the new code and writing blog posts. The time all these things take even if the size of the new code is less than 100 lines is substantial.
But, I do have challenges working with their ACL structure -- which is backed by their database -- and specifically would love to be able to update users/groups separately from tag groups and both of those separately from ACL rules and I can't help but wonder if the fact that this feature doesn't exist relates to the fact that the database doesn't normalize these concepts into separate "tables" separated by "foreign keys".
Unrelated.
We just haven't worked on it much. A bunch of other stuff has been a higher priority. But better ACL management will happen.
And if they got on ok with the json file, maybe you should update towards the features of a complicated RDBMS not being worth the cost?
> MySQL (or PostgreSQL) would come next. I’m not particularly familiar with anything MySQL post 1998, but I’m sure it would work. The HA story for open source databases is somewhat surprising, though: you can either have traditional lagging replicas, or commit to no-primary-replica clusters that have very surprising transaction semantics. I wasn’t excited about trying to design a stable API or good network graph calculations on top of those semantics. CockroachDB looked very promising, and indeed still does! But it’s relatively new for a database and I was a little concerned about getting attached to features in a fresh DBMS that would be hard to migrate away from if we needed to.
Seriously.. consider this.. they are worried about the transaction semantics of using db replicas and instead CHOSE SQL LITE!
(It's sqlite, or SQLite if you're being pedantic; it's not "SQL LITE").
It sounds like you're essentially arguing that people should always use the most robust data store available regardless of cost or requirements. Their needs were served just fine by a json file for quite some time — if that's all you need, then maintaining a full Posgres cluster is the "crazy" solution actually.
There is incredible value in incrementally adding complexity only when needed.
And, completely apart from the database, it's a really valid point - not a joke at all.
When someone says, "It's 2022. Why are we still arguing about X?" or some such, it means that they haven't been able to persuade the rest of the world about X on the merits of their argument, and so they want to short-circuit the argument by appearing "modern" or "up to date" or "on the right side of history". They want to win by persuading the other side to give up in embarrassment, not by virtue of the quality of their position. (In their own eyes, their position is probably self-evidently the only right and moral one, but that doesn't tell us much.) It's a way of trying to win by default, and it deserves nothing other than being regarded as "you lose by default".
Well, calling yourself "AnimalMuppet" you kind of set yourself up for dude-dom.
> Using a text file and ... > It seems like something you'd do as a proof of concept
I mean, using a text file for the proof of concept is exactly what happened. And then it grew too quickly and had to get off of it eventually, but we always knew that. We were just amazed how long that worked. It survived much longer than we'd thought.
Hmm, I can’t quite articulate why, but I would. Something about having the courage to be different.
If a single JSON file worked at the time, then that's clearly all they needed. It certainly hasn't seemed to limit their success.
i feel your pain dave. resume driven development and FAANG envy seem to be the dominant characteristics of far too many folks in the industry these days.
For me, it's not so much my fellow developers -- half my age, but quite bright enough to realise the benefits of simplicity -- but that the vendors of the tools our corporation has decided to use have in turn decided to implement them not just as a spaghetti-reinforced mudball, but as a spaghetti-reinforced mudball of spaghetti-reinforced mudballs.
(That these SRMB^2s are then just part of a larger spaghetti-reinforced mudball of SRMB^2s, i.e a spaghetti-reinforced mudball cubed, is of course not the vendors' fault but corporate. But then, aren't corporate systems always SRMBs? So if at least the components were simple, the end result would be only an SRM to the power of one, not three... Sigh.)
I'd say this could easily be a great engineering decision if it meets the needs of an organisation in the moment. The thing with any engineering decision is that it involves a bunch of trade-offs – doing something that you know will hit a scaling wall can be worth it, if it can get you moving faster. There will never be a perfect solution, and in these decisions I find that simplicity is often a massively under-rated benefit.
I'd even venture to say that I've seen more organisational failures as a result of optimistic over-engineering than I have optimistic under-engineering. That includes many of my own projects – ones where I've gone "oh I guess I'll do it properly this time" and then found myself six months down the line throwing out a ton of code, schema, or infrastructure I spent ages on because some requirements changed or some assumptions didn't pan out. It's hard to remember that lesson sometimes and I've been doing this for 15 years.
It's like a company I used to work for where the whole product was driven by a massive semi-structured graph database – but instead of being implemented like that it was a big ball of MongoDB JSON documents glued together with PHP by someone who was more of a domain expert than an engineer. Often pretty sketchy to work on, but an engineering decision that was a success – in the sense that it allowed that product to exist at all where it otherwise might not have.
why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these decisions?
Sign me up as both an engineer and a user. It is extraordinarily refreshing to hear "we tried a bunch of different stuff, some of it was dumb, and this is where we're at now."
This might actually be a really good filter for which types of engineers are good fit for their company. Because my read was very different than yours, as I looked at the decision lineage and thought to myself I could see myself making every one of those decisions. I of course can't say I would've with the same problems and constraints they had, just that I know I've considered similar solutions.
The big difference might be, on the third iteration I tend to land on embedding raft directly into the backend, because lots of the problems I look at tend to benefit from a in-order log that is fully replicated to a quorum of servers. But based on the post and efficiency they're going for backing up the WAL on sqlite may make alot of sense.
Actually litestream looks like it might have some fairly similar properties, so might be something else I'd like to add to my toolbox. Although I wonder what sort of replication lag there might be to replicas when using s3 as the stable store.
A good engineer can build all sorts of surprising things. A senior one knows why they should reserve that for only the special bits. I like reading the first two types of articles -- we frequently give tech talks ourselves -- but for making boring systems weird, I get flashbacks of consultantware/resumeware/NIH that took ages to weed out. I try to give talks on say GPUs for weirdly shaped problems, not on why we celebrated when we finally had time to rip out mongo from where it didn't belong :)
I've worked with Raft a lot in my career. I wrote the original etcd Raft implementation (sorry!) [2] and even wrote a visualization explaining Raft[3]. It's a great tool when you need really high durability guarantees but I'm not convinced it gives you better availability guarantees than a single node that can quickly be recovered.
[1]: https://mtlynch.io/litestream/ [2]: https://github.com/goraft/raft [3]: http://thesecretlivesofdata.com/raft/
I know they say this isn’t April fools but I’m not sure I would put it past the engineering team. This can’t be real.
“We looked at real solutions like cockroach but nobody could be bothered to rtfm”
On my old team, our product supported Oracle DB (along with some other databases), but on one on the team really was an oracle expert, and we would frequently run into questions that we didn't really know how to answer. We would have dropped it if we could.
It's easy to say "rtfm" but doing this doesn't make you an expert in the system. And as I bet many NH readers know, there's a big difference between "I read the quickstart guide and got a DB working in an afternoon" and "We've hit an error an error that the manual says nothing about, but our DB expert saw this once, 7 years ago, and remembers how to recover"
Yup, this rings true. Once worked on a large Cassandra deployment – holding people's bank accounts, so, ya know, somewhat critical – and we eventually had to get one of the authors of Cassandra onto our on-call team. Even he was stumped by some of the bugs and failure modes. (And, incidentally, the word 'anticompaction' is still enough to trigger my PTSD.)
The notion that the documentation is an accurate and exhaustive explanation of every single thing about a product is a junior engineer mistake that no one with rough ops experience would say. Nor, for that matter, anyone who's written a database that's in use (as I unfortunately have), or really any other piece of software.
And the same goes for the notion that "everyone should go with MySQL or Postgres for everything because they are Proper Databases that are scalable and won't go down". If I were being charitable, I'd say it's a 'no one got fired for buying IBM' argument; if I were being honest, I'd say it puts me in mind of 'MongoDB is webscale' (https://www.youtube.com/watch?v=b2F-DItXtZs). I don't know why bad engineers are so terrified of thinking for themselves specifically – well, generally, but especially – when it comes to databases. They aren't magical contraptions. They are just portable filesystem wrappers.
> I'm sorry, but what the fuck? Just writing the blog posts alone might have consumed more work-hours than what it would have taken to set a database up in the first place.
> Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision. It seems like something you'd do as a proof of concept or side project. It's interesting that they're blogging about this, as if they're proud of it.
> This is their 3rd DB Migration, something that I prefer to avoid at all costs. I guess they have different priorities and values. But I'm just confused, why would an engineer want to join a company that is making these decisions?