Hacker News Reader

Comment by KwisaksHaderach | Hacker News Reader

The obvious candidates were MySQL (or one of its renamed variants given who bought it) or PostgreSQL, but several of us on the team have operational experience running these databases and didn’t enjoy the prospect of wrestling with the ops overhead of making live replication work and behave well. Other databases like CockroachDB looked very tempting, but we had zero experience with it. And we didn’t want to lock ourselves into a cloud provider with a managed product like Spanner.

What about managed mysql/posgresql? no lock-in and installing them locally is trivial.

View on HN

Managed DBs have tremendous lock in. Just try migrating off RDS with zero downtime. You can't, because they've "managed" your ability to configure external replicas. Then built a whole brittle data migration service that probably won't work for your DB.

they've already had to migrate twice due to questionable technology decisions. having to migrate again due to needing to change cloud providers seems a lot less likely than having to migrate again because their outside the box technology choice didn't pan out.

"Questionable technology decisions". You're trying to dunk, but I don't think you understand where the hoop is. Their technology decisions have panned out Tailscale well. We should all be so fortunate. This isn't Twitter with the "fail whale"; the only reason you know about any of this stuff is because they wrote about it. They ran their entire service with a JSON file backend for 18 months, and switched from it to etcd without you even knowing about it.

i really don't know a ton about this product or team but it sounds like if they had used aurora mysql or aurora postgres in the first place then there would be nothing to write a blog post about because it would've just worked and kept working. they say they want to avoid vendor lock-in but if the vendor became a real issue they'd be doing their first migration instead of being on v3 already. additionally, their bespoke solution relies on s3, which is also a vendor-specific technology, so it seems like they haven't avoided vendor lock in? i've seen many cases of developers doing more work to avoid vendor lock-in than it would take to replatform if it ended up being a necessity, and this really feels like that looking at it from the outside. i'd understand this better if mysql or postgres couldn't solve their problem, but that is not the case here, and i can't wrap my head around a company who is ok with their devs reinventing a very good wheel 3 times when the obvious choice would've worked fine the first time. it seems like they are successful in spite of these decisions, not because of these decisions.

https://mcfunley.com/choose-boring-technology

You should start by learning more about the product, and then tell them they should use Aurora for all their backing store.

The s3 api is rapidly becoming a distributed storage standard. Building your product around it is hardly lock-in these days.

AWS does offer one (very reliable) implementation but they’re very definitely not a monopoly.

>> it seems like they are successful in spite of these decisions, not because of these decisions

That is my conclusion, too.

The thing about startup decisions is: most of them are "wrong". Or they start right and become wrong later. Successful startups aren't successful in spite of their wrong decisions, they're successful because they can change them very quickly (then write a blog post about it and get more customers).

There's also a strong correlation between people who look at things from weird angles and also build good products. Why are you surprised the people who invented an entirely new way of doing VPNs also don't cargo cult database storages?

it's not cargo culting if it works, and they even say in the article that mysql or postgres would've worked

maybe next they will stop cargo culting operating systems and switch to SerenityOS?

They're talking about running many instances of MySQL locally, not hooking all their systems up to RDS.

no they aren't

Tailscale’s coordination server, our “control plane”, has become known as CONTROL. It’s currently a single Go process on a single VM.

source: https://tailscale.com/blog/an-unlikely-database-migration/

Sounds like you need to do a lot more research before commenting on this? It's pretty easy to find in the Litestream docs that it replicates to S3, Google Cloud Storage, Azure, and other options even including SFTP. In fact by off-loading the storage integration details to Litestream, the Tailscale people now get seamless storage vendor independence almost for free.

This is what a really smart and future-proof solution looks like.

or they could have used mysql or postgres hosted by aws, gcp, azure, etc? i would put money on this not being the last time they change databases

Again: this is an infrastructure component, running on many machines internally. It's not a Rails app. We use SQLite in very similar circumstances across hundreds of machines. Having all those machines schlepping all their reads back to RDS would not only be untenably slow, but it would also make the whole system less stable. I don't think you've really thought through the design at all, and you should before making comments like these.

I'd put money on this not being the last database change too! But not for the same reasons you would.

Unless I'm misunderstanding something, it seems to me that Tailscale's controller, discussed in the previous blog post [1], is like a typical CRUD API server, currently only running on one machine (they're not using Litestream's work-in-progress live read replicas), not something widely distributed like Fly's service discovery infrastructure.

I'm not joining the "just use RDS" crowd, but it looks to me like using a managed DB service like RDS would have been a reasonable decision for this application.

[1]: https://tailscale.com/blog/an-unlikely-database-migration/

I'll take that money quite easily ;-)

They're not questionable because they didn't work, they're questionable because they were so difficult to run that they had to migrate twice.

You could absolutely run a web site using Brainfuck without any failures or any customers realising it. Doesn't mean it's not a questionable decision!

The etcd solution was highly questionable. Actually, a prime example for reinventing the wheel, but worse.

Choosing a technology that supports your business needs and growth, and that also lets you easily migrate to something else once it stops being suitable (or your requirements change, or you discover a way in which the tech doesn't work for you, or whatever), doesn't sound like a questionable technology decision, it sounds like a great one.

I'm having a really difficult time migrating my managed postgresql instance in Google Cloud just to get it to a newer version. I'm down to my last option: shut down everything, export, re-import somewhere else again.

I did briefly have logical replication working, and so had my near zero downtime solution in hand. But the replication broke after a few days because I delayed cutting over, and then I couldn't get it going again. I couldn't find out how to debug this either.

The lesson I've learned here is that I'll be avoiding managed DB products whenever I can. It takes away control that turns out I sometimes really need.

If you install it locally, that is pretty much the definition of an unmanaged database. Other than by the user of course.

You can use a managed version of the database in production, and an "unmanaged" local Docker image during development.

Reply on news.ycombinator.com