ckwang

The "back" proxy was the initial setup with SmartStack, so we went with that for minimal viable first steps. We wanted to make incremental changes, changing as little as possible, for this migration so we could monitor for correctness and performance at every step. The eventual plan is to run Envoy as a front proxy for ingress, and maybe even Envoy <-> Envoy everywhere, where we have Envoy as both a front and back proxy on every service deployment (instance, container, etc.)

We route all outbound traffic from internal services through Envoy, even if the destination isn't running Envoy. We don't have envoy running as a "front" proxy right now, i.e. our L4 setup isn't Envoy <-> Envoy, it's Envoy -> service directly. An example of this is the DB layer - traffic going to our DBs from services goes through Envoy service-side but Envoy isn't running on our DB instances.

Yes! HAProxy is a terrific piece of tech, and has been awesome for our use cases so far. We do quite a bit with it for our main ingress routing and it was basically flawless as our data plane in SmartStack.

I'm really excited about what we're building out for next year and can't wait to share as well. Feel free to reach out on reddit (u/wangofchung) or directly at courtney.wang@reddit.com for more in-depth discussion!

That's pretty close to our next immediate steps as well :)

We run one proxy per machine, even when there are multiple services running. The proxy is just an abstraction to the downstream dependencies. Even if there are multiple services per machine, they can still reach downstream services via the same proxy path.

Per-service proxy deployments are a bit complex for the infrastructure but provide a nice abstraction for the service and service developers themselves. The configuration scheme is indeed daunting, which is what we're hoping Envoy and its xDS APIs + centralized configs can help us solve for developer teams.

We're actually looking to put Envoy in front of the redesign stack at some point in the near future! The major services backing the redesign can be isolated into a few smaller pieces, and we'd like to have Envoy be a routing layer that can abstract this for the central browser client as we evolve the backend.

We're currently evaluating the pieces that comprise Istio, both within Kubernetes and outside of it in our existing infrastructure.

We didn't do so immediately because we did not want to immediately update all of our technology at once and felt that a piece-wise migration would be both the least disruptive to our infrastructure and safest. I think of Istio as like Smartstack in that it's not actually a complete "thing" so much as a suite of technologies that can be individually evaluated and deployed. It's very easy to fall into the trap of wanting to do everything at once, and we opted to make small progressive steps for this initiative.

We have not yet put Envoy in our main HTTP ingress path, but internally we have designs and implementation paths ready to go, and it's definitely being considered for public ingress traffic. As we noted in the last "teaser" section of the post we'd really like to leverage Envoy's routing functionality to facilitate migrating client-facing APIs in the backend without affecting frontend interfaces.

Our HAProxy layer that routes ingress traffic to the core backend infrastructure has considerable routing logic that can be moved to Envoy and then further extended. We'd love to explore that path in the coming months.

Let me start by saying: The gotchas that we've experienced have been far eclipsed by the benefits.

Aside from what what discussed, there weren't many things with Envoy specifically. We did have a few minor issues with the Thrift tooling, e.g.

https://github.com/envoyproxy/envoy/commit/a3c744294bca2cbf6...

but as the above indicates, they were resolved _very_ quickly.

The most important thing when making a transition like this is to have as much monitoring and observability as possible without the new tech. We were able to quickly identify and respond to issues we had with Envoy based on existing application and system instrumentation that weren't directly provided by Envoy, along with the vigilance of our engineering team.

1. We are considering Istio! This is especially true for our Kubernetes environment. We are already planning to deploy Pilot for the first iteration of our control plane in our non-K8s environment, so the other pieces that comprise Istio is a natural place for us to continue exploring.

2. We have not containerized prior to Envoy. We're running application binaries provisioned with Puppet on EC2 for most of our infrastructure still.

Hello! Blog post writer here. Happy to answer any questions.

It's odd to me too. They had a great deal of pre-order information, something that most other server teams that face crazy launches don't ever have. I would guess that they could at least estimate the load to an order of magnitude. The fact that single player can't even be accessed is absurd, something that your comment on sharding hits on. The server teams at Blizzard should also have had some experience with this in their WoW trials and errors.

I'm curious as to why you feel that way. Why is "harshing on frats" as you put it not a negative act? Also, how does the reddit post not apply to the original issue here?

"LOL @ Phi Delta Toolbag."

I would think that's pretty offensive to anyone who's in any sort of "* Phi Delta *" fraternity. In a thread that touches on the offensiveness of stereotypes and subsequent alienation, the irony of both the comment and your LOL reaction should be pointed out. As far as "thoughtful comment" goes, there certainly wasn't much thought put into the phrase "Phi Delta Toolbag".

That's actually my lecture that ClassMetric broke down in the blog post. From a (first time) instructor's perspective, I don't see the tool as a distraction to students. I'm not punishing students for using it, and students have another way to communicate to me that they don't understand something in lecture. I only lectured for one hour a week, the TAs spent the majority of the time with the students. In my case, Class Metric allowed me to touch on the big ideas of the week during my lecture, measure confusion on topics, and figure out how TAs should structure their discussions and labs to be the most useful to the most students. The message posting system also allowed TAs to participate directly during lecture as well, answering questions and addressing confusion as it appeared so students could better keep up with the lecture material. Yes, it may be distracting to have knowledge coming from two channels, but students can still refer back to that channel later when they are reviewing the whole lecture material.