mattrobenolt

Sure, there's definitely risk. I'm not asserting that it's literally 0 chance. But this risk of this is also tied up with other things that leverage this proxy. So it's not adding a new dependency or a new point of failure. If this has a problem, we'll also have problems talking to other services in our network.

And for what it's worth, I've definitely fucked this up in the past and caused downtime as a result of a setup like this. The pros still outweigh the cons in practice.

This is a trusted, internal, private network. The only one who could do this is the application itself, or something rogue on our network. If something were running rogue on our network, there'd be worse things it could get access to.

Yeah, we could do something like this as well if we cared. We are easily caching 95+% of our active data in a small amount of disk space. It's not that valuable for us to have a complete replica of the full data set.

Ironically, in our case, our availability is very core to our business. In this exact scenario, if S3 would have blocked our processing pipeline because it was down, that means we couldn't have alerted users as reliably that they were in fact, having issues because of S3 being down as well. So in our case, this is massively important to us and worth any additional risk that may be introduced.

Technically yes. But we're pretty accustomed to this level of risk. For something like this, the pros far outweigh the cons involved. Yeah, it could fail. The maintenance overhead of this is absolutely minimal and took a handful of hours to have tested and in production.

Also worth noting, that this isn't really a single point of failure as a system wide thing. It'd only be a single point of failure on that single node. So if haproxy decided to explode, only that one machine would have a problem momentarily, while the process got started back up with our process manager.

The worst case scenario is a human error where we ship a bad config and break everything.

Fortunately, I am top engineer and am on-call for what I implement and rely on. :) This is also why I build things in a way that don't add more risk to production infrastructure. If you'd read the post, you'd see that this is a strict improvement without risk on our side from introducing a new dependency.

This is correct assessment. HAProxy is running on localhost, and it strictly falls back to hitting public S3 directly if our cache is down.

It's running on localhost to each server. So the failure event here is that somehow haproxy process would explode with the rest of the server being fine. It's much much more likely that a whole machine will die instead or a network issue between machines, etc.

Sorry, and by reliable, I explicitly mean, the network connection from our datacenter to halfway across the country hiccuped more than I liked and was slower than I liked. Not reliability of the service itself.

This is mostly unrelated to our original goals, which was reliability and performance. This wasn't to protect ourselves in the event of S3 going down, the timing just worked out that it saved us during that as well while also accomplishing the original goals.

That's another problem that'd require much more work than just a few hours of hacking around. :)

I'll have to look into this. I wasn't aware. Either way, I don't think that'd replace our current setup since the original intent wasn't to increase our availability. But good to know!

Yeah, one of our goals was to not add a new/weaker point of failure.

We gracefully fall back to S3 directly if our cache server is down without a hiccup. So there is no operational overhead of this additional cog. If the server has a failure, we'd go back to slightly degraded performance by talking across the country until we brought it back online.

I have tons of experience with Varnish and a long history there. Varnish is really bad for this since it's memory only. I wanted to use 750GB of disk space, not the 32GB of RAM we had. 750GB of RAM is significantly more expensive.

And in our case, performance between reading some bytes from disk vs memory isn't significant. A disk seek is still many many orders of magnitude faster than a round trip to Amazon.

With that said, Varnish does offer the ability to use mmaped files, but the performance is really appalling out of the box, and just not worth it. Varnish is way better if you want strictly in memory cache.

Another benefit of nginx is the cache won't be dumped if the process restarts, unlike Varnish.

Definitely. If our dataset was absolutely massive and we couldn't hold a reasonable amount on disk, it'd make more sense. Fortunately, we are getting a 90+% hit ratio out of a very small amount of space relative to the size of our bucket.

And yeah, we have 0 resiliance for write data here. Again, fortunately, we can afford this tradeoff since the amount of uploads is significantly lower and much less critical for us.

It directs traffic conditionally based on if our cache server is up or not, falling back to S3 directly. In theory, we could use nginx for this, but nginx isn't as good as a generic load balancer since it doesn't have as rich of insight into the status of upstreams. Also yes, the nice benefit of being able to do alerting based on uptimes and whatnot. We already have haproxy running here, so it was just adding another frontend to support this.

Or we can do what we did and save money instead and gain the guarantees. Plus achieve our original goal of performance by bringing the bytes inside our datacenter instead of going out through public internet.

S3 bucket replication is a bit flawed, and doesn't buy us anything on top of what we implemented. And would cost double for storage. Plus more complex application logic which I wanted to avoid. Tools like haproxy are pretty good at this.

With S3 replication, you still have a primary/replica setup in which only one of them can accept writes, but you can accept reads from both. So we'd gain HA between multiple regions, but we wouldn't solve our original goals: speed. The round trip to S3 was too slow for us.

In our case, we don't need to ever revalidate. We store things forever since our file blobs are immutable.

That's DataDog. :)

As mentioned, we are saving a lot in S3 transfer cost. :)

This would unfortunately be slower, since it's not within our datacenter and on our network, and significantly more expensive. Probably at least 100x more expensive for the bandwidth than S3 by itself.

I guess this is just phrased a bit oddly. It's just a factor in helping us scale by requiring us to use our hardware more efficiently. Not sure I'd say it's horizontally or vertically. We've just made each unit of work cheaper to help things along.

Nope. That's me. :)

What's your PayPal address?

> So right now when we have a need to send a stacktrace (error or panic) we use Raven-Go's NewStacktrace to create a stacktrace and attach it to the packet we send up. In the UI I see the exception with surrounding lines.

Yeah, so raven-go will check for the source files on disk first. You must keep your source files along with the binary. :)

> I'm wondering what the reasoning is behind suggesting I upload source artifacts to you as part of a release. The blog post even specifically calls out Go, which is what the majority of our application is written in. So from my understanding, uploading artifacts would allow the Sentry UI to show me exactly where we did a Sentry Dump?

For Go, you ultimately compile your application into a binary. This binary doesn't have the source code, but it does retain references to the filenames and line number in the binary's metadata. What this means is, if you send along a stack trace to us, we can tell you that information, but we can't show you what the actual line of code is, or what the surrounding code is. By uploading the sources to us explicitly, then baking in the release version into your compiled binary, we can map those up and show you a stack trace that includes source code. This is relatively the same idea with any modern JavaScript. Typically a project is compiled down into a minified version, or run through a transpiler, etc, and the original sources aren't accessible to us. So it's then necessary to upload to us so we can map things up.

> On the same topic, our current deploy flow makes it a bit hard to upload specific code changes per release. It would be great if Sentry could get the code from Github directly (assuming that I login and give you access to the repo)

We have plans for VCS integrations in the future (GitHub included) that would likely work with releases so we can just use the working directory at a current sha.

Just our realtime system, and we run some asynchronous tasks with Go now, specifying to replace a few things that were done by Celery previously.

See this shitty thing I wrote to bridge the gap: https://github.com/mattrobenolt/go-celery :)

For me, it felt comfortable writing Go, considering I've been mostly writing Python for the past 8 years. So the transition was a lot more natural, compared to using Scala or Erlang or anything else.

Again, this is my subjective opinion, and this is why we chose to use Go for our new stuff instead of something different.

I assure you, I built this in a week. :)