mattrobenolt

:see_no_evil: This is an obvious next step for us.

I went down the Prisma path investigating JavaScript ORMs, and... I have opinions. :)

I feel that what we have would be useful within a Prisma context, but given their complexity and how Prisma works, it's likely not very practical with their "engine".

Netlify uses Deno as it's runtime. :) https://github.com/planetscale/f1-championship-stats/blob/ma...

So we work there just fine.

We didn't talk much about this yet, but the underlying tech for this will help you within Lambda too reduce latency even over normal MySQL.

This specifically is targeting environments where a MySQL client isn't able to run.

There are, in my opinion, a few other concrete benefits outside of performance too that I didn't touch on. I plan on touching on them more in my follow up blog post though.

A lot of this for me/us at this point is still theoretical since we don't have tons of application yet. Serverless was an easy first target since they require HTTP as the transport. The real experiments will come when we say, put together a Python or Go driver based on these APIs and compare side by side to a native MySQL driver.

Our beta for this is pretty stable. The only real instabilities are around the underlying APIs we use, which is part of why we're not ready to document it just yet.

But the underlying tech is exactly the same as we use for handling traditional MySQL connections, so there isn't anything to fear.

Great question! Author of a lot of the infrastructure surrounding this here.

I plan on following up with a more technical deep dive on some of these aspects, but it's quite a bit hard to do a 1:1 comparison.

While, obviously, HTTP also uses TCP, I'm going to assume you're asking more about the binary MySQL protocol vs HTTP.

On the surface, yes, HTTP is going to have more overhead with headers and the other aspects that come with HTTP. Also, in this case, the payload is JSON, both on the request and response, which is going to be slightly more bulky than the payloads over the MySQL protocol.

So where things get interesting is the real world performance implications. Networks and CPUs are really fast. HTTP has been extremely scrutinized and JSON as well. While they are admittedly not the most efficient standards, they are used so heavily and parsers are heavily optimized for these protocols.

Mixing in modern TLS 1.3 with modern ciphers, which is something you're very unlikely to get with a traditional MySQL client, we can achieve a much faster first connection time. Pairing with modern ciphers that are demanded for TLS 1.3, and the transport itself can be significantly faster than a slower cipher. This isn't the best comparison since typically you're using MySQL without TLS, but when talking to a service provider like us, we require it for obvious reasons.

Next, with HTTP, we get to leverage compression. In our case, we can use browser level native compression with gzip, or if using something server side, we support compressions like snappy. Combine something like gzip or snappy with HTTP/2 now, and given that the bulk of a query result is typically the result itself, and not the protocol, compression can make up a decent amount of the difference. Again, I'm being hand wavy since every query pattern and results are going to be wildly different based on your data, so it's not anything fair to say "this is x% more or less efficient."

And lastly, with HTTP, especially HTTP/2, we can multiplex connections. So similar to the gains with TLS 1.3, you can do many many concurrent database sessions across 1 actual TCP connection with HTTP/2. And similarly to TLS 1.3, the cost here is in connection timings, and management of a connection pool. You don't need a pool of connections on the client, so your startup times can be reduced.

As a stretch goal, HTTP/3 (with QUIC) is in the crosshairs, which should eliminate some more transport cost since it uses UDP rather than TCP. My hunch is all of this combined together, an optimized driver leveraging HTTP/3 might beat out a MySQL client overall. Time will tell though!

So there's no simple answer here, everything has tradeoffs. :)

Hey, developer of Sentry here. We're submitting a patch that prevents showing any settings on this DEBUG page.

I'd like to mention that we never suggest running Sentry in DEBUG mode, nor do we document how to do this. Sentry does use Django, so it's pretty easy to put pieces together and figure out how to do it.

So while we can help obfuscate this specific issue, running in DEBUG mode always has the potential to surface insecure information due to the nature of it.

Our patch to suppress this information entirely can be found here: https://github.com/getsentry/sentry/pull/9516

Why would you want to lazy load it?

What kind of couch do you have that can fit 6 people? Looking for a new couch.

This is equivalent to shipping bad application code that takes everything down. Except the config is only a handful of lines of code and will very likely never change again. Also, we don't blindly roll out changes cluster wide for things like this without testing explicitly on staging or test nodes.

SoftLayer. All internal bandwidth is free. Public network is a fixed allocation of bytes per server, then overages.

This is bad for stuff like this because nginx doesn't re-resolve DNS records after process startup. So if an IP address behind the hostname changes, things will just hard stop working. Using it explicitly as a variable coerces nginx into actually resolving DNS regularly to pick up changes like a normal client.

Is paragraph 2 of the post not upfront enough? I clearly stated our goals of the project, and the fact that this also saves us through the incident was a great side effect. But I never set out to mitigate S3 failures like this.

Also, should note, it's about a 98% savings in bandwidth, but cost savings explicitly was 70% since we have to factor in the cost of running this new server.

Thank you. <3

This doesn't solve our performance issues that I originally set out to address.

Ah, I can't comment about that since we obviously don't use it, but in theory, yes.

We do this automatically already with HAProxy. So we don't even have to change our application.

I'm afraid to know how much that costs.

Disagree. Considering the original goal had nothing to do with expecting S3 to go down, it just happened to be super useful during this incident. We get more more benefits even without S3 breaking.

We even go a step further, and our blobs are 100% content addressable. :) So caching is super easy for us.

It's running on localhost, so it's not it's own machine. It's local to the servers running the application code.

Yes. That was, in fact, the entire point of the blog post.

To be more clear, there are other things we could do here if we didn't inherently trust our network and the things running there.

Good question. Authorization is passed along upstream to S3, but we don't re-check authorization when serving a cache hit. In our case, this is a fine tradeoff since our network is private and trusted.

You should actually read the blog post. We are strictly increasing availability in addition to S3's already amazing availability. Not adding another point of failure and assuming we're better. In fact, I literally assume I'm shitty, so I build defenses so my fuck ups don't cause any issues.

> You can't add a new thing without adding a new point of failure. Every point is a point of failure.

Correct, but it's an existing process. So you're right, we could ship a blatantly bad config.

> Who deploys the thing?

We do, humans, yes. We can definitely screw up a config.

> Automated deployment?

We tend to do blue/green deploys on critical pieces of infrastructure just to sanity check it. We might even pull a node out of production, test on a staging server, etc.

> Wait, was the local proxy load tested?

Yes. The load we need for this case is not even close to significant.

> Was it load tested when one of your data centers is down and everything is doing 30% more work?

Yes, it's literally just a proxy to S3 doing no additional work. For our traffic, the load is not a concern. Especially since it's running on every machine, it's distributed pretty well. A single box cannot overload our haproxy process compared to the CPU needed to run the Python application itself.

> Can you tell I used to work in monitoring? Maybe I just have PTSD now. :P

DataDog, it's pretty dope. It gives us lots of super good insight into all of these things and is what alerted us because haproxy reported S3 down in the first place. It'd also tell is the moment a process like this crashes, etc.

Yep, it's definitely possible to go this route as well. We just didn't have to.

You missed the major point where we're strictly improving and not thinking we're more available. In fact, I assume that we're less available, hence why we have a fallback from our local to S3 directly since I assume our single server is going to die more often than S3.