Hacker News Reader

Comment by frellus | Hacker News Reader

To be fair, there are going to be plenty of critical leaks coming out of Twitter with a 50% cutting of staff. That's so many whistleblowers, my backyard is completely leaf free now. Frankly, if Musk cuts $1B from the run rate and there isn't a bad outage, a lot of people will look pretty stupid.

To be fair, I don't know if Twitter is overspending on infrastructure, but I would guess there are always places to streamline. You could frankly save a ton of money by re-negotiating commits with Google.

Also, $1B to me shows how insane running a large business in the cloud can be. This is sort of where Facebook/Meta got it right, IMHO - run infrastructure on-prem, at scale, in a super-efficient "everything fails" sort of way.

View on HN

I had a friend that worked for a massive company in the Valley which I'll leave unnamed, that told me about them just leaving servers running hot with no throughput for months at a time. So, I guess it's possible. I doubt his estimate was pulled out of a hat (but I could be wrong, or the article could just be pulling that out of a hat/using an unreliable source.)

I've taken over AWS and Azure accounts before that were paying tens of thousands for unattached disks, load balancers, big servers that were doing very little that could have just been on a burstable VM. Massive messes that in some cases took months to clean up. So, I can believe there could be a massive bit of cost savings available if it were poorly managed. I've never seen the scale of a company like Twitter, but I really doubt there isn't some dead weight. I know storage is a massive cost, and he has stated he was going to purge accounts that were unused over a certain time frame.

One of our smallest customers had a VM for each application server process. CPU occupancy alone was rarely over 20%.

Easily cut 70% of cloud costs.

Yep, same here; as a (well paid) hobby, I help smaller companies lower their software bills. When they use 'cloud hosting' instead of a vps or something, that's usually the first point where I can cut 80%+. It is crazy how many things are running there that are not used or needed in almost every company I see.

Usually there are software issues (most commonly, no, not enough or plainly wrong db indexes being the #1 top scorer) which drive rds bills to scary height for no reason at all, but the low hanging fruit is unused cloud 'stuff'.

It's been a fun 'arms race' of sorts between hard-drive manufacturers and FAANG-esque coders. I have a friend that works for a very big hard-drive company. So, think very large data storage at scale. Well, they discovered that a fair bit of their spin-disk machines would get to the max rotation lifetimes with little reader-head arm movement. AKA, they were just run for months at a time without ever being really used. This was after the inherent 'shut off if nothing has been written to you in XXX sec.' code that runs on every machine.

They had some digging about, and it turned out that most of the FAANG guys would just write a test bit every XXX-1 sec. to keep the disk spinning. As it improved their internal metrics on 'up time' or whatever else they thought was important. So silly.

So, the manufacturer then wrote code to deal with that code to try to save the bearings. And soon enough that code was discovered, and more complicated code was made to keep the disk spinning no matter what.

And back and forth until my buddy retired and stopped caring about this strange little 'arms race'. I suppose it's still going on though.

This is fascinating, and deserves its own article. People do the darndest things.

If I can get my buddy to write one, I'll make sure to post it to HN. Thanks for the idea, it may be enough to get him to write something up.

Same here, 70% cut in costs. If efficiency isn't a goal and there isn't a strong push for controlling costs, they just keep going up.

I don't know if 1B is a reasonable goal, but Twitter's previous leadership was so incompetent that I wouldn't be surprised. Maybe even more is possible.

How do you find clients for this hobby? What sort of message or phrasing do you use?

Could you explain how db indexes are costs having no reason at all?

I assume tluyben2 means that if the db's index is bad the query has to do a much more expensive scan of the database. By adding correct indices this can be fixed.

That is indeed what I meant; with very little effort and not having much knowledge about database internals, you can save a ton of money. It won’t be perfect that way, but goes a long way vs no indices at all. So that’s why ‘no reason’. Aka very low hanging fruit.

Sorry for not being clear and thanks for explaining.

No to mention adding an index is practically without risk, the outcome of queries is the same except probably faster.

Not true.

An index isn't free so you can impact the write performance, depending on the workload and usecases slower writes could have more of an impact than faster reads.

But even in a read heavy application it's possible that your databases query planner is "tricked" into using a specific index which can actually be slower than another (or even none) so you should always measure the impact that adding an index can have on your application.

Yes it could be slower, but it’s very unlikely to actually break the application.

Not true, as always it depends.

Tell me you haven't used spanner without telling me...

Everyone is “overspending” on infrastructure. There is always money to save the question comes down to where you want to expend your engineering effort and what risk levels you want to take.

“Make fewer features, focus on spending less” is a valid executive decision, it’s just a matter of priority. Twitter seemingly being feature complete and losing lots of money most years makes doing this now a decent decision.

Spending too much like that is generally a symptom of not paying attention and noticing the waste, it happens from time to time. Gets to feel normal, then you notice and correct.

I’ve been mostly responsible for cloud spend a number of times and had several events of cutting spending by 20% or more over the years.

I've had the same experience at startups. Usually it's time to make features, but eventually the spend gets high enough to justify spending a couple weeks of engineer time on it to cut out waste. Same thing happened when I worked a large internet company, just on a different scale.

AFAIK Twitter's serving infra is mostly on-prem and they use cloud mostly for analytics.

https://blog.twitter.com/engineering/en_us/topics/infrastruc... (2017 tho)

I don’t think the 1B/year shows issue on cloud pricing. It’s instead bloat of twitter’s setup. That’s way in excess of what should reasonably be spent. I suspect there’s a lot of bad coding practice going on. Of which the bulk is data analytics.

AWS profit margin is something like 35%. I think some of their infra is there, so that might be an upper bound?

Do they use AWS though?

https://blog.twitter.com/engineering/en_us/topics/infrastruc...

if Musk cuts $1B from the run rate and there isn't a bad outage, a lot of people will look pretty stupid.

A then-current Twitter employee told one of the New York Times podcasts that Twitter's infra looks overbuilt because the teams focus primarily on reliability, with new features presumably coming second.

Feels like Twitter is now joining the "move fast and break things" club.

They barely run anything in the cloud, they’ve been operating their own data centers & infrastructure for over a decade. So they’re doing the same thing as Meta.

> Frankly, if Musk cuts $1B from the run rate and there isn't a bad outage, a lot of people will look pretty stupid.

Is this a good way of reasoning?

If somebody said "frankly, if Bob stops wearing his seatbelt and doesn't die, a lot of people will look pretty stupid" we'd push back. Now the opposite is also not true, where if things go fine Musk is still an idiot. Only the people who know the internals of the system can really say with confidence. One thing I am sure of is that Musk is not one of those people.

Except Twitter going down is not as serious as death. The answer to “How much $ is it worth to get an extra 9 of reliability?” is never “Infinite”. So, I agree with OP, if they can save a big chunk of their opex with a negligible decrease in reliability (measured by something like, how much user churn are we seeing as a result of the downtime), then it’s a good call.

I don’t know enough about Twitter’s internals to guess whether this is a good call. A bit off-topic, but having worked at Meta I think it would probably not be a good call there. Infra engineers are already directly incentivized to find fractional-percentage wins, because the savings to the company are often greater than their total comp, and that’s a pretty straightforward way to justify a high performance rating.

For Twitter, outages are serious business and definitely a threat to the organism.

As users on the outside maybe we don't care so much, but internally Twitter has invested tremendous amounts of energy and brain power to make the service resilient and scalable over the past 12+ years.

When Twitter is down, it was taken quite seriously by those who worked there. I'm sure they did the blameless postmortem culture, the whole none yards.

Too much downtime and people will stop using it and / or migrate to a competitor, and then the service will be dead.

The analogy you're using is binary (seat belt on or off), whereas an infrastructure cut is more like reducing your thermostat by a couple of degrees, then concluding that no one froze to death, therefore it was an acceptable cut. Many parts of Europe will be running this experiment, in fact.

more like, cutting off heating to the building, telling everyone to wear sweaters, and just wait and see if anyone has a serious medical emergency.

I am quite puzzled, at such scale with low seasonality what are the reasons they haven't moved to their own infra? Managing and hiring people to do so would result in big decrease in cost compared to the cloud.

Twitter is mostly on their own infra. 1 billion is 25% of total revenue. I think they are close to that spend with cloud but it’s more like 750m if I were to venture a guess.

> This is sort of where Facebook/Meta got it right, IMHO - run infrastructure on-prem, at scale, in a super-efficient "everything fails" sort of way.

It's fairly ignorant to make this claim without realizing that Twitter also runs the majority of their stuff on-prem as well.

Reply on news.ycombinator.com