bjacokes

Sure, one example I remember off the top of my head is a bank that sometimes returned duplicate transaction data, so an engineer had called ramda.uniq on the transaction array. Transactions are nested objects and slow to compare, so when you find an account with 100,000 transactions... kaboom. Some scenarios are more subtle, but a common theme is that the amount of data in an account can vary by many orders of magnitude.

You hit the nail on the head here. When N different API requests simultaneously time out – all because a ramda.uniq call in one of them received an array of 100,000 nested objects – it's easy to make a spot code fix, but harder to systematically prevent it from happening in the future. There aren't really linters for "bad event loop blockage". Code reviews are the main tool we have, but you'd be surprised what sorts of logic can trickily block the event loop. For API reliability and development velocity in the short-term, by far the easiest approach was to throw more infrastructure at the problem.

We do use Go for almost all of our other services, and there are an increasing number of integrations written in Python. But we're still using and investing in our Node integrations code for the foreseeable future, and this was an important step for simplifying our infrastructure.

We certainly hope the tooling and rollout process in the post were instructive for anyone using Node, even if their stacks were pristine from day 1 and never need this sort of complex migration :)

It was chosen about 6 years ago when the product was first being developed, so most of us on the engineering team weren't around when the decision was made. The main choice we're making at this point is: what's the impact and ROI of a language migration vs getting Node to work as well as we can?

Our integrations are primarily written in Node, which was the original language used for everything at Plaid. Almost all of those original services (except for integrations) have been migrated to Go or Python at this point. We've standardized on gRPC as our wire format, so we stayed consistent and used gRPC in Node.

With perfect hindsight, it's a fair point that all the pros and cons could net out to another language being best for our integrations. Integrations are the largest and most quickly-changing codebase at Plaid, so such a migration would be a massive undertaking. We definitely didn't want to block scalability improvements on doing a language migration.

We thread knowledge of whether a data pull was initiated by the API or by our cron-style service into our load-balancing layer, so this ends up being pretty straight-forward.

There are a couple of reasons that the legacy scaling model was viable for us. As mentioned in the post, only 1/10 of our traffic was from the API, which gave us a roundabout way to scale by diverting resources. And it's only viable to use this model of scaling when the business value of a request is high – we were originally quite happy to spin up more containers when we reached our scaling limit. That's the pragmatic reason why we were processing one request per container.

In terms of what issues caused us to move away from parallelism in the first place, it was all the CPU-bound stuff that you might expect: ReDoS-style issues, post-processing arrays in very large edge cases, programmer error, etc.

I'm a little confused by this approach; are there any non-essential containers in your suggested architecture? This sounds like the watchdog container is just a parent process that launches a bunch of subprocesses, which is definitely a workable solution, although not the one we decided to use. If there are primitives for an essential container to inspect container state and relaunch other containers in the task, that'd be great to know about.

Yeah, I mentioned this in a comment in a different thread. The duplicate containers in the task definition need to be marked as "essential" in CloudFormation to make sure our capacity doesn't degrade on container exits, and this means that one container exiting will also exit other containers in the same task. So we have a bleed-over effect where OOM in one request could cause N-1 other requests to fail.

The essential vs non-essential container designation is a little confusing. The standard use case for multi-container tasks seems to be that all containers are marked as essential, i.e. they essentially represent different services that are operating in concert on the same machine. This is definitely not the situation we're in, where each container is totally independent.

So it seems like we'd be a perfect use case for non-essential containers. However, (1) at least one container _must_ be marked as essential, and (2) non-essential containers which exit don't get restarted or replaced. This means we would still have a limited bleed-over effect (if the essential container exits, the other ones do too), and more importantly, we can't guarantee that our capacity will be robust to process exits.

If we handled N concurrent requests per process, a crash would definitely have the bleed-over effect we're trying to avoid, where a problem in a single request caused the other N-1 to fail. "Crash" meaning primarily OOM or unhandled rejections (which we exit on). And yes, the other big problem is blocking the event loop with an inefficient regexp, massive sort operation, clumsy use of Ramda, etc. This sort of blockage causes our ECS health checks to fail, so the container eventually gets killed.

"Never returning" is an interesting problem in Node. We have gRPC request timeouts, so the request fails on the client side after a set amount of time. Our Node server gets a cancellation event when this happens, but there's no guarantee that the in-flight request ever stops processing or gives up its resources, even if it's not blocking the event loop. E.g. our request handler could have a while(true) loop that continually does an async network request, and even though each gRPC request eventually times out, we would eventually have a swarm of zombie while(true) loops that are operating. To address this problem, we thread a lightweight context object into our framework which can essentially call an isCancelled() method. Before doing most low-level async operations (e.g. network requests), we check isCancelled() and throw if the gRPC request has timed out.

The integrations are all written in Typescript. As mentioned in another comment, we've attempted multiple processes per container and multiple containers per task. Right now, we see the bigger win as being able to fix "all of the above" and run multiple requests per Node process, but it'll take some legwork to get there :)

When we start processing a request, we take the JS Promise that represents the result of that operation and put it into an array in our internal "server state" object. When the server gets a shutdown request, it basically awaits Promise.all(this.inFlightRequests) before exiting. Depending on the LB/routing layer on top of your service, you might need to do this in a loop in case work comes in after you do your first await. In other languages you can join a thread pool, wait on a wait group, or use whatever other synchronization tools are provided.

Note that there's an ECS_CONTAINER_STOP_TIMEOUT parameter in ECS that sets a hard upper limit on how long containers have to exit before getting SIGKILLed, and it defaults to 30 seconds. If you want to allow requests to drain for longer than that, you'll need to update that parameter.

(For services with fast request processing time, the draining process is often a lot simpler. You can just route traffic away from the server, then wait a short period of time before telling the process to exit. It's not quite as precise, but works well for many use cases and requires no bookkeeping.)

Glad you pointed this out! We could have gone into a lot more detail on the reasons that we've arrived at our current system architecture, but it would've distracted from the root problem we were solving in the post. Happy to go into it a bit here.

Node event loop blockages are the primary reason we have so many processes running. We have enough integrations and iterate on them quickly enough that our infrastructure essentially treats them as untrusted/breakable. We want to avoid ReDoS-style bugs from affecting more than the current request, so we handle one request per process. A little inelegant, but we've still been able to horizontally scale the system, and frankly the extra infrastructure cost hasn't been enough to be worth the effort to change it.

To get around the start-task rate limit, we've tried running multiple identical containers per ECS task. However, they need to be marked as "essential" in CloudFormation to make sure our capacity doesn't degrade on container exits, and this means that one container exiting will also exit other containers in the same task.

Multiple processes per container is another interesting approach. We've used Node subprocesses in the past, but we found them tricky for reasons that are unrelated to deploy speed.

One thing we've really liked about rolling our own approach is that we decide when to declare a deploy complete. ECS is pretty conservative about not declaring a deploy complete until the final container has finished draining, which can take minutes for some of our requests. With our fast deploys, we declare a deploy complete when the final container running old code stops accepting new requests, which is significantly sooner. This makes follow-on deploys and rollbacks much smoother.

In a way you're right, which is why we are exploring kubernetes. At the same time, ECS has worked really well for us as we've grown to this point. It's worth noting that we have some other motivating reasons to switch off ECS beyond deploy limitations, such as poor observability of ECS agent logs.

It's inevitable that early systems are going to bend and break at various points of the scaling process, and the interesting question is how to navigate that – fight fires for awhile or fix things immediately, hack together a stopgap solution or do the full fix, etc. We thought ours might be an interesting data point to share.

When you sell shares, cost basis is reported to the IRS by the brokerage. The parent comment suggests that RH always reports the FIFO cost basis, i.e. price of the first shares you bought as opposed to the last shares. There are places on tax forms where you can adjust your cost basis up or down from the one reported by your brokerage, but (without doing any research into it) I doubt that "I don't want to use my brokerage's cost basis method" is a valid reason to use those.

Dividends and buybacks both indicate that the company is willing to return cash to shareholders. One big difference is that dividends are typically a long-term commitment to returning cash (unless it's a "special dividend"), whereas buybacks are more ad-hoc. The other difference is that buybacks imply an opinion that the stock is undervalued, whereas dividends are agnostic about valuation. In most cases, buybacks seem to be motivated by the former difference more than the latter. However, some companies like Berkshire are more principled about valuation driving buyback decisions.

I would disagree with your point about stock issuance suggesting belief in the company. Issuing stock is usually a negative signal that not only is your company struggling with cash flow, but it can't issue debt at a reasonable cost. (Startups issue equity because it is difficult to borrow at such a risky stage of the company, but more established companies like Tesla issuing equity is usually a sign of difficulties with cash flow.) Read up on cost of equity vs cost of debt for more info.

There are fewer shares, but the book value of the company goes down as well because it is spending cash to buy the shares. So it's not necessarily true that the price should go up.

While I love reading about logical fallacies, I don't see how I've committed one.

I never said building infrastructure would convince everyone to bike. Rather, the government can (and often does) move society as a whole in one direction using money as a lever. Do taxes on gas or alcohol prevent everyone from driving or drinking? No, but they move the needle, just like building cycling infrastructure does.

And I never cited the "10% increase" as an important metric for the success of cycling infrastructure, you did. None of my arguments is about the absolute number of people cycling, although others have pointed out that biking is likely underfunded relative to the number of people who use the infrastructure. I just cited the increasing number of people on bikes as a counterpoint to your claim that "no matter what people say, they don't want to cycle to work".

Do you have a citation for 50% of the SFMTA budget going to cycling infrastructure? This sounds demonstrably incorrect given the fact that they run the bus and LRV systems.

(Also incorrect is that people "don't want to cycle to work", I see literally hundreds of them every morning in SF, and it's growing every year.)

There are many arguments in favor of cycling infrastructure, but unfortunately I'm versed enough in anti-bike comment threads to recognize someone who won't be convinced :) But here goes...

* bikes do less damage to road than cars, and therefore the maintenance cost of bike lanes is lower than car lanes, which require frequent paving

* bikes are important as a "last mile" transportation option for public transit commuters. densely-placed bike share stations improve the feasibility of public transit for people whose homes or offices are 1-2 miles from a train station.

* bikes are far smaller than cars and so you can push more traffic through a given width of road than with cars. this is especially relevant if you ever get to a downtown stretch where one car turning right at a pedestrian crossing causes an entire green light to be wasted on the rest of the cars (basically every light along Market Street in SF). cars are just wide and clumsy when driving in dense downtowns.

* the government frequently subsidizes behaviors that are considered virtuous (or conversely, imposes "sin taxes"), and also subsidizes services that benefit lower-income citizens. both of those apply here. car parking in downtowns can run $40+ for a day. bike share and bus commutes are $2-5 and keep people active.

I could go on, but again, I don't think your mind will be changed much.

Another acquisition model is what Goose Island did when they sold to ABI. As I understand it, they outsourced production of their less differentiated offerings like Honkers Ale and 312, but continue to run their brewpub and make their Bourbon County Stout and Belgian beers like Matilda and Sofie. You see this in some other industries, where a company will spin off or sell businesses that aren't its "secret sauce".

Did she bother changing her voice on the phone vs. when she met Mary in person? For familiar voices, it's often pretty easy to know who's speaking even when they change their pitch or accent, but maybe it was practical to assume that Mary wouldn't be able to notice a voice she'd only heard once on the phone. Obviously this isn't a key part of the exploit since she could've always gotten someone else to do the phone conversation, just wondering how careful she needs to be with those sorts of details to avoid something going wrong.

As others have pointed out, the rideshares are subsidized in many ways. Monetarily, not just by VCs, but also by drivers who don't get benefits or salary guarantees. Road infrastructure provided by the city that isn't fully supported by gas taxes. Externalities like constantly-blocked bike lanes and congestion that inconvenience others in the city.

$4 doesn't get you far in SF, but if you do pay that rate to go across town, just know that you're not paying your fair share to support the driver, maintain the roads, and are probably inconveniencing other drivers and cyclists more than if you were using a bus. Which is fine — but if you then go and argue that this is a reason to never support public transportation, you're just buying into the Lyft/Uber marketing.

The small student population at Olin affects the founder rate in a more pertinent way than sample variance: because they only need to admit 100 students, this group of students can be of a higher average quality than if they needed to accept 1000+ students like Stanford/MIT. As an analogy, if YC were to 10x the number of startups they invested in per batch, they'd certainly expect their average return to decrease, because the last company they accept will be less likely to succeed than the first or tenth.

If you really wanted to measure how Olin's teaching compares to that of MIT/Stanford, vis-a-vis startup founding success, you could have the other schools "draft" a team of 100 CS students at the beginning of their freshman year, and measure the founding success rate of only those 100 students.

Here are some of the positive qualities I would expect from a competitive programmer compared to the general CS population early in their career:

- knowledgeable at algorithms and data structures

- good at analyzing correctness and edge cases, even on simple non-algorithmic problems (e.g. FizzBuzz)

- accustomed to working hard and learning new concepts. this attribute is not specific to competitive programmer – for example, I'd expect similarly from an open source contributor – but it's higher than for a typical college student.

Some negatives I'd expect, which are fixable over the course of the course of their career:

- over-confidence in code, under-testing

- less skilled in OOP, coding style, version control systems, as well as web development or systems code (unless they have specific previous work in these areas)

- sometimes looking down on gruntwork/rote as beneath them (like the view that pure mathematicians have towards applied math or statistics)

I think that list of positive attributes often outweighs the potential negatives, especially during an internship or in the first year or two of someone's career. After that, I would expect many non-competitive programmers to have picked up on some of those advantages (code correctness, learning new concepts).

I've tried to steer my own interviews away from algorithms (especially DP) and focusing more on giving problems that are relatively straightforward, while still being complicated enough that someone has to write precise code and identify/fix a few edge cases.

It seems irrational that people are hung up on the time spent shopping, but not time spent cooking.

When I tried Blue Apron, I spent an hour cooking a meal for two. That's 30 minutes of work per serving.

When I cook with my own ingredients, I spend an hour shopping and an hour cooking, then make 8 servings and refrigerate/freeze the leftovers. That's 15 minutes of work per serving, and it's significantly cheaper.

I agree that Blue Apron makes more sense if you don't have access to a good grocery store, or if you don't want to invest time in finding recipes.

Note that Google and Facebook have effectively the same level of founder control, albeit with different implementation details. Google even had to issue a new share class in 2014 so that the company could issue more shares without the founders giving up voting control. Snap's approach seems like a more robust long-term way of telling investors that they don't have a say in things.

One phrase that I've noticed (over)used in NFL broadcasts is "give credit". After a win, a star player will often "give credit" to the other team, their defense, lesser-known players, etc. It seems to be a decent way to act humble when you're the center of attention, without saying "humbled".

Check out https://github.com/pkg/errors

Assuming a robust job market, you won't have much exposure to a company's success/failure other than through equity, so I don't think diversification makes sense as a reason to avoid equity compensation.

I could see this being a valid argument if there were systemic risk to the industry that could have a simultaneous effect on your equity and salary compensation. e.g. if you're a truck driver and get equity in your company, then automated trucks could be disastrous and you'd want to diversify away from the equity.

A corollary to your 20-30% "uncertainty discount" is that the expected value of a pre-IPO offer should be higher than an offer from a publicly traded company. It's the same reason that there is an excess return or risk premium (again, in expectation) for investing in stocks over bonds.

This makes some assumptions about risk-aversion and rationality – it's possible that the latter is an optimistic assumption, given the amount of koolaid-drinking inherent in joining startups. But as long as you're telling your friend to discount his startup offers, and articles like this are being written, I think it's logical to think that the pendulum will swing towards mid- and late-stage startups building in such a premium to their offers, especially if they're serious about getting good talent.

Someone buying a $1.5m condo only has to put up $300k for the mortgage, so it's not correct to say that the alternative is to just put $1.5m into the stock market: the buyer often doesn't have that much.

Also, it's more precise to use the S&P dividend yield (2%) or 10-year Treasury yield (2.4%) to gauge how much someone with $1.5m could spend in rent, assuming they put all their interest into rent. They'd be renting something at more like $3k/month.

Now take the fact that home prices are, at the higher end, about 25x the yearly rent of the same place. So this person with $1.5m is essentially renting a place worth $900k, whereas the buyer (who perhaps only has $500k) is living in a nicer place that cost $1.5m. And that's not considering the fact that rents could rise further.

There are taxes and maintenance to consider, the risk of putting a lot of money into a volatile asset, and being tied to one place. Financial leverage in general is risky. But it's not a clear-cut decision like most people try to make it.

Maybe this is a better way to look at it.

Candidate: "We're going to build a wall between us and Mexico. It'll cost us $7 billion."

Electorate: "Umm. That's a lot of money. Let's do something else with it."

If that's how people would react, they should react the same way even if we're getting the money from Mexico. Money is fungible, and as much as our hearts enjoy poetic justice, it shouldn't matter where it's coming from.

There are a couple of situations where we would still want it built, even if its cost exceeded what we were willing to pay ourselves. One is if this is a cooperative, non-zero-sum situation: if Mexico were getting some value from the wall, they might be willing to pay some of the cost, beyond what we can extract via tariffs and seized remittances. The other is if Mexico is terrible at negotiations, which is likely what Trump is assuming.