Comment by partiallypro | Hacker News Reader

Comment by partiallypro | Hacker News Reader

Parent Full thread

I had a friend that worked for a massive company in the Valley which I'll leave unnamed, that told me about them just leaving servers running hot with no throughput for months at a time. So, I guess it's possible. I doubt his estimate was pulled out of a hat (but I could be wrong, or the article could just be pulling that out of a hat/using an unreliable source.)

I've taken over AWS and Azure accounts before that were paying tens of thousands for unattached disks, load balancers, big servers that were doing very little that could have just been on a burstable VM. Massive messes that in some cases took months to clean up. So, I can believe there could be a massive bit of cost savings available if it were poorly managed. I've never seen the scale of a company like Twitter, but I really doubt there isn't some dead weight. I know storage is a massive cost, and he has stated he was going to purge accounts that were unused over a certain time frame.

One of our smallest customers had a VM for each application server process. CPU occupancy alone was rarely over 20%.

Easily cut 70% of cloud costs.

Yep, same here; as a (well paid) hobby, I help smaller companies lower their software bills. When they use 'cloud hosting' instead of a vps or something, that's usually the first point where I can cut 80%+. It is crazy how many things are running there that are not used or needed in almost every company I see.

Usually there are software issues (most commonly, no, not enough or plainly wrong db indexes being the #1 top scorer) which drive rds bills to scary height for no reason at all, but the low hanging fruit is unused cloud 'stuff'.

Could you explain how db indexes are costs having no reason at all?

I assume tluyben2 means that if the db's index is bad the query has to do a much more expensive scan of the database. By adding correct indices this can be fixed.

That is indeed what I meant; with very little effort and not having much knowledge about database internals, you can save a ton of money. It won’t be perfect that way, but goes a long way vs no indices at all. So that’s why ‘no reason’. Aka very low hanging fruit.

Sorry for not being clear and thanks for explaining.

No to mention adding an index is practically without risk, the outcome of queries is the same except probably faster.

Not true, as always it depends.

Not true.

An index isn't free so you can impact the write performance, depending on the workload and usecases slower writes could have more of an impact than faster reads.

But even in a read heavy application it's possible that your databases query planner is "tricked" into using a specific index which can actually be slower than another (or even none) so you should always measure the impact that adding an index can have on your application.

Yes it could be slower, but it’s very unlikely to actually break the application.

Tell me you haven't used spanner without telling me...

Same here, 70% cut in costs. If efficiency isn't a goal and there isn't a strong push for controlling costs, they just keep going up.

I don't know if 1B is a reasonable goal, but Twitter's previous leadership was so incompetent that I wouldn't be surprised. Maybe even more is possible.

It's been a fun 'arms race' of sorts between hard-drive manufacturers and FAANG-esque coders. I have a friend that works for a very big hard-drive company. So, think very large data storage at scale. Well, they discovered that a fair bit of their spin-disk machines would get to the max rotation lifetimes with little reader-head arm movement. AKA, they were just run for months at a time without ever being really used. This was after the inherent 'shut off if nothing has been written to you in XXX sec.' code that runs on every machine.

They had some digging about, and it turned out that most of the FAANG guys would just write a test bit every XXX-1 sec. to keep the disk spinning. As it improved their internal metrics on 'up time' or whatever else they thought was important. So silly.

So, the manufacturer then wrote code to deal with that code to try to save the bearings. And soon enough that code was discovered, and more complicated code was made to keep the disk spinning no matter what.

And back and forth until my buddy retired and stopped caring about this strange little 'arms race'. I suppose it's still going on though.

This is fascinating, and deserves its own article. People do the darndest things.

If I can get my buddy to write one, I'll make sure to post it to HN. Thanks for the idea, it may be enough to get him to write something up.

How do you find clients for this hobby? What sort of message or phrasing do you use?

Reply on news.ycombinator.com