Comment by suresk | Hacker News Reader

Comment by suresk | Hacker News Reader

Parent Full thread

It's always funny to read about crazy overspending on infrastructure and I have a few funny stories myself, but this was far from their biggest problem. For one, we don't know how big those costs were or how much time it would take to wind some of it down.

I absolutely hate waste and was always proud of myself when I found ways to shave a thousand here and there off our AWS bill, but it was a bit depressing when I went through the thought exercise of "What would it be like for the company if I managed to get everything onto the AWS free tier?" And then compared that to what it would be like for the company if we found a way to make 1 or 2% more in revenue.

Early on, the math heavily favors worrying less about infra spend and more about product enhancements that can build revenue.

(Although often reducing the size of your infra for other reasons - usually around complexity for ops or ease of development can also have the side benefit of reducing cost and it seems worthwhile in those cases.)

>Early on, the math heavily favors worrying less about infra spend and more about product enhancements that can build revenue.

YES! This is VERY true

It is a careful balance. At my last startup we knew we had scale technical debt in our SW. Not something we could fix by tossing HW at the issue. It was all great until we landed that major customer that blew past our supported scale. 6 months of engineering work and a complete stop of feature process. A more careful consideration of the tradeoffs when making them would have saved a huge headache, but no one wanted to look at the issue preferring to ignore it.

> Not something we could fix by tossing HW at the issue.

I'm curious to learn why, if there's any more you can share. What about vertical scaling, i.e. bigger machines instead of more of them? Can't that go pretty far?

Shipping unit was a VM to a customer. Tossing more CPU & RAM at the VM would hit general issues of how VMware allocates CPU to a guest. Within the VM we had containers for each sub-unit of the tool. For some part of the system we made the deployment model more than one VM, distributing more containers to the other VMs. However for a key part of the design there was an in-memory DB and we just it the scale limits of the number of elements that could be handled that way without a rethink of how and what we stored.

For everyone that will say "well why..."

1. real products need a generally available shipping unit (SKU) when your customers are enterprise / telco types.

2. for item 1, what is the lowest common thing this type of customer would take - a VM they can run on their existing virtualization setups. the reality of K8s, containers in this type of customers at that point was near zero.

3. there was thoughts about a SKU of a HW appliance, ie, beefy server and just toss CPU/RAM in it and ship that. shipping some stupid HW appliance that we controlled would just be a distraction and a waste of money also requiring 2 shipping artifacts, 2 test setups, etc. not to mention the need to sort for support for HW failure (you can contract with Dell for example to have them label their Kit as your appliance).

Shipping a VM makes a lot of sense. I wish there were better off-the-shelf tools for packaging up a web application into an easy-to-deploy, self-managing VM, without trying to stuff the typical cloud complexity (e.g. Kubernetes) into a box.

I don't understand the issues with using a bigger VM, but I have no reason to doubt you.

Reply on news.ycombinator.com