Hacker News Reader

> Anytime you perform a DB / Redis write aren't you back to waiting 70ms-300ms depending on where in the world you are?

But that's always going to be the case surely with Postgres? If you're running your one DO box config in say US East, every request is going to have that 70-300ms response time for people not in US East region.

With Fly, I think what they're saying is if you are doing writes to DB then you are going to get no optimisation over having one box. However, if you are just doing reads, then you are going to get a lot of optimisation from having it closer to the user.

If you had a simple blog for example, reading blog articles would be fast as it could serve locally, but if you created a blog article it would be 'slower' as you would have to write to the DB. Given for most blogs are 99.99% reading articles you are going to get a significant speedup.

The problem is though that many applications do writes regardless of what the user is doing (analytics/audit logs and that kind of thing).

I think what Fly needs (maybe they have this already) is a specialised message queue which sits in the same region as your servers and doesn't require waiting for global sync. You could then put many write operations through that, and only have to wait for global sync when you need to show something in the UI immediately.

> But that's always going to be the case surely with Postgres? If you're running your one DO box config in say US East, every request is going to have that 70-300ms response time for people not in US East region.

Absolutely, but in DO's case you'd be paying $10 / month instead of $186 on Fly since you're accepting the situation for what it is (you have 1 server in 1 location).

> However, if you are just doing reads, then you are going to get a lot of optimisation from having it closer to the user.

Yes but this also comes at an additional cost I think. Their shared CPU + 2 GB PG cluster is $27.40 a month. Based on their pricing page it's not clear on how this would be priced across multiple regions and read-replicas only but if it's like their app servers then it's multiplied by the number of regions, so if you had 6 of them across the world it would be $164.40, then you have your $186 for your app servers for $350.40 a month and you still have high latency writes. We haven't gotten to Redis costs either. Keep in mind this is for a 1 CPU + 2 GB set up too, the costs get multiplied if you need more compute power.

> The problem is though that many applications do writes regardless of what the user is doing (analytics/audit logs and that kind of thing).

I think there's a lot of cases where you'll have a good amount of writes independent of audit lots. For example a chat system, or leaving comments on anything. There's a lot of scenarios where writes are common and this isn't a problem related to DB scaling. Each one of those writes will cause the user to wait a long time (location dependent) since your controller action will wait for your database to return before it sends an HTTP response back to the user. But I do agree there's tons of opportunities to take advantage of read-replicas for reads.

> I think there's a lot of cases where you'll have a good amount of writes independent of audit lots. For example a chat system, or leaving comments on anything. There's a lot of scenarios where writes are common and this isn't a problem related to DB scaling. Each one of those writes will cause the user to wait a long time (location dependent) since your controller action will wait for your database to return before it sends an HTTP response back to the user. But I do agree there's tons of opportunities to take advantage of read-replicas for reads.

Yes definitely but many applications are 99.9% read, and stuff like chat doesn't really require instant <100ms responses. It really enables a load more performance for those usecases.

For many of those B2C sites $350/month would be nothing to save a load of latency and increase conversions. It's obviously not a solution for every single web app out there.

I'm right there with you. You need roughly 1 data center per continent to get fast enough response times. The 10-30ms you save in transit time from the edge to a central location isn't worth the complexity or cost.

This is why we don't say "edge". Most apps work best with 3-6 regions. Apps that benefit from ~20ms websocket round trips might use more.

How can I ping the regions without having to register?

Like you point out, compute and network are but one part of the infrastructure equation for stateful apps, one which Fly is adept at. Fly's database story will only improve with time (and I believe they are already looking to partner with other database SaaS vendors), but till then, read-replicas are a pretty resourceful solution.

What Fly really excels at is the dev-ex of deploying apps globally (they don't really compete with VPS providers, that's a different market, imo). There's no mucking around with deployment, orchestration, configuration (and to an extent monitoring) to distribute apps over Fly's infrastructure.

> If you have let's say 6 instances spread across the globe with 1 dedicated CPU + 2 GB of memory each that's $31 x 6 ($186) just for your apps.

Agree. They need to work on their pricing tiers. It is a steep jump from $2/mo with 256M RAM + 0.05vCPU (?) to $31/mo for 1vCPU + 2G RAM.

Unless they are a hobby company, they should increase the prices. The money is in the head, not in the tail. If someone is balking at spending $200/mo on the lowest level, they are simply not a customer.

> Unless they are a hobby company, they should increase the prices. The money is in the head, not in the tail. If someone is balking at spending $200/mo on the lowest level, they are simply not a customer.

It's not balking at the $200 by itself.

It's $200 with the message that I should now have a globally distributed app with low latency because that's what Fly talks about all over their site.

The balk is because all of that goes away as soon as you use Postgres or Redis (which is a ton of apps -- especially Rails). It makes Fly come off as being dishonest because this is a huge thing that's not mentioned anywhere on the pages that try to sell you their service.

It feels like they are leading you into using their platform, you hook everything up, finally deploy your app and then realize that a large percentage of your customers are still getting +200-300ms location related response times because they happen to be outside the location where you're hosting Postgres / Redis. This gets even worse if you heavily invested in Websockets and bet everything on having low latency but still have high latency. This is something that could make your break your business all because of mixed messaging on Fly's site.

I don't think they are doing this on purpose btw. I just think it's easy to amplify a problem so it looks really bad and scary so you can come in and offer a solution. This is a common sales tactic. I just wish they were more open about what happens when using common tech like Postgres and Redis.

The article you're responding to is exactly about Postgres. Here's Redis: https://fly.io/blog/last-mile-redis/

Here's a boring Rails app that uses Postgres: https://fly-global-rails.fly.dev/

And here's more Postgres:

https://fly.io/docs/getting-started/multi-region-databases/

https://fly.io/blog/globally-distributed-postgres/

Rails and websockets are not very good, Phoenix and websockets work great though.

Half of everything their website talks about is Postgres exactly because that is the main source of issues.

If you can make requests without writing even once, then Fly can be a great system (especially because you probably don’t need their larger instances in those cases).

I think they do a fairly good job of telling you that it’s not for you if you want to write to the DB on every request.

I agree with you that I cannot see any use-case for that, which is why I’m not using Fly.

> I think they do a fairly good job of telling you that it’s not for you if you want to write to the DB on every request.

The biggest thing they say on their home page is:

    > Deploy App Servers
    > Close to Your Users
    > Run your full stack apps (and databases!) all over the world. No ops required.

Technically it's true because "databases" includes reading, but they don't mention anything about how this model falls apart as soon as you want to perform a database write with Postgres.

The entire top area of their home page is optimized to make you think "my real world app is going to be deployed close to users and look how easy it is to get going, I just run 2 commands and it's multi-region deployed".

Right below that they have a whole section dedicated to Postgres but they don't mention anything about writes being limited. Instead they talk about highly available clusters and specifically throw in read replicas to leave the burden on the user to infer what they really mean is that writes aren't multi-region and will involve high latency based on the user's location.

It's a conflict of messaging. If this service is meant for folks who aren't hardcore into ops they might not even know to think about database writes on their own because the messaging is so geared towards "we do all of this for you, just run 2 commands and you're deployed".

> I think they do a fairly good job of telling you that it’s not for you if you want to write to the DB on every request.

I didn't find any of this on their home page and if it happens to be mentioned a few times in passing within blog posts or deep in their documentation this feels more like misdirection than helpfulness. It feels more like other vendors in sales where they have a sales page to optimize conversion rates but then somewhere on page 76 of their terms and conditions there's fine print that says "btw, except for when you want to do anything like DB writes with a SQL database, none of these benefits apply" and now if they get questioned they point to this condition and say you should have read it.

I Googled for "fly.io postgres writes" and only found 1 article related to postgres writes, but it's really unfortunate because the blog post misleads folks immediately.

It's at https://fly.io/blog/globally-distributed-postgres/

I took a screenshot of it in case it gets edit here: https://imgur.com/a/eY6EJ0e

In the first non-bolded paragraph they have:

> We won’t bury the lede: we’re going to talk about how you can deploy a standard CRUD application with globally-replicated Postgres, for both reads and writes, using standard tools and a simple Fly feature.

Right off the bat they are saying they solved the problem of having a globally distributed postgres set up for both reads and writes using Fly. They even go as far as bolding "for both reads and writes" so that if someone skims this page that might be enough for them to think "awesome, ok Fly is perfect for me" and now head to their page to sign up because all they wanted to know is if that's possible or not, the details can be filled in later.

But then later down in the article, below the fold they write:

> But these schemes break down when users do things that update the database. It's easy to stream updates from a single writer to a bunch of replicas. But once writes can land on multiple instances, mass hysteria! Distributed writes are hard.

So now they contradict their premise that they solved reading and writing in a distributed way and go back to say on second thought, nevermind, we can't handle distributed writes because it's hard. Then later on in the article near the bottom they mention you should avoid using postgres if you have a lot of writes and use cockroachdb instead.

What kind of messaging is that?

I appreciate what they're trying to do but they send mixed signals around solving a problem they haven't solved which also happens to be tied into their biggest selling point (globally distributed deployments so that end users have low latency responses).

> So now they contradict their premise that they solved reading and writing in a distributed way and go back to say on second thought, nevermind, we can't handle distributed writes because it's hard.

Fwiw, even Postgres-centered "real-time" platforms like Supabase do not do distributed writes (but they sure can benefit from distributed reads).

> I appreciate what they're trying to do but they send mixed signals around solving a problem they haven't solved which also happens to be tied into their biggest selling point (globally distributed deployments so that end users have low latency responses).

I get where you are coming from, but I'd imagine Supabase on Fly is instantly faster than its default single-region deploys on AWS. Fly's CEO, Kurt Mackey, is an investor in Supabase, so let's see how that partnership pans out.

Fly's primary/read replica setup is as good as it gets without complicating matters any further. My money would be on Fly.io to rope in CoakroachDB, PlanetScale, YugaByte, CitusData as solution partners, but that's easier said than done.

> I took a screenshot of it in case it gets edit here: https://imgur.com/a/eY6EJ0e

You could use archive.is (https://archive.is/rlxsS) or web.archive.org/save, too (:

Increase their prices? Their prices are already completely insane. A dedicated 8-core CPU with 64GB RAM for $558.16/mo? And that's not even bare metal, you're sharing all of the other resources of the server. How is it possible that netcup can give you two more cores for under 50€ [0]? Or Hetzner a dedicated server, not just CPUs, for a similar price [1]? Housing servers is expensive without your own datacenter? No, it's not, it's cheaper than ever [2]. What is Fly doing with their servers? Or is that all just markup? It seems their customer is somebody who doesn't care at all about money or somebody who is completely unaware of how much servers actually cost.

[0] https://www.netcup.eu/bestellen/produkt.php?produkt=2632

[1] https://www.hetzner.com/sb?ram_from=5&ram_to=8&cpu_from=9000...

[2] https://dc6-cz.translate.goog/cenik-server-housingu/?_x_tr_s...

A few things here:

1. We pay basically the same prices for dedicated hardware you would. Hetzner is cheap, in part, because they run in a single facility and optimize for price. We run in 22 locations and optimize for "cheap enough for most app devs".

2. All our servers are AMD Epyc CPUs – meaning more expensive than consumer CPUs. Our margin on hardware is almost exactly 70%, if that helps.

We're not competing with Hetzner, Hetzner is a great f'n company. If they do what you need we'd rather you use Hetzner. I could give you some TCO nonsense but I won't. I do think we're a pretty good value though!

The money is in the head, not in the tail. If you cannot afford $200 on a distributed application, go rent a server at hetzner

> If someone is balking at spending $200/mo on the lowest level, they are simply not a customer.

>> Ultimately, we want to accomplish two things:

>> It should scale well, from small mostly text based web apps to large, intense apps that handle a lot of data. You should be able to build a CDN on Fly.io without worrying too much about your bandwidth costs. You should also be able to run a hobby project for close to free.

>> Transparency and predictability are good. People should be able to glance at pricing, do minimal mental math, and judge how well it works for their app.

from: https://fly.io/blog/we-cut-bandwidth-prices-go-nuts/

Btw, technology adoption begins at the "lowest level", if Geoffery Moore and Clayton Christensen are to be believed: https://archive.is/42NS6#selection-586.0-586.1

Eric Schmidt disagrees. You might have heard about the company that adopted that as the approach. It is called Google.

Oh, thanks for clearing that up. I thought you might have been talking about Novell.

Novell was trying not to be the expensive kid on the block. We know how well it worked out for them.