kelp

I'd originally posted this here: https://lobste.rs/s/surdxc/is_billion_dollar_worth_server_ly...

But cross posting in case it's interesting to this audience.

Over the past few years of my career, I was responsible for over $20M/year in physical infra spend. Colocation, network backbone, etc. And then 2 companies that were 100% cloud with over $20M/year in spend.

When I was doing the physical infra, my team was managing roughly 75 racks of servers in 4 US datacenters, 2 on each cost, and an N+2 network backbone connecting them together. That roughly $20M/year counts both OpEx and CapEx, but not engineering costs. I haven’t done this in about 3 years, but for 6+ years in a row, I’d model out the physical infra costs vs AWS prices, at 3 year reserved pricing. Our infra always came out about 40% cheaper than buying from AWS for as apples to apples as I could get. Now I would model this with savings plan, and probably bake in some of what I know about the discounts you can get when you’re willing to sign a multi-year commit.

That said, cost is not the only factor. Now bear in mind, my perspective is not 1 server, or 1 instance. It’s single-digit thousands. But here are a few tradeoffs to consider:

Do you have the staff / skillset to manage physical datacenters and a network? In my experience you don’t need a huge team to be successful at this. I think I could do the above $20M/year, 75 rack scale, with 4-8 of the right people. Maybe even less. But you do have to be able to hire and retain those people. We also ended up having 1-2 people who did nothing but vendor management and logistics.

Is your workload predictable? This is a key consideration. If you have a steady or highly predictable workload, owning your own equipment is almost always more cost-effective, even when considering that 4-8 person team you need to operate it at the scale I’ve done it at. But if you need new servers in a hurry, well, you basically can’t get them. It takes 6-8 weeks to get a rack built and then you have to have it shipped, installed, bolted down etc. All this takes scheduling and logistics. So you have to do substantial planning. That said, these days I also regularly run into issues where the big 3 cloud providers don’t have the gear either, and we have to work directly with them for capacity planning. So this problem doesn’t go away completely, once your scale is substantial enough it gets worse again, even with Cloud.

If your workload is NOT predictable, or you have crazy fast growth. Deploying mostly or all cloud can make huge sense. Your tradeoff is you pay more, but you get a lot of agility for the privilege.

Network costs are absolutely egregious on the cloud. Especially AWS. I’m not talking about a 2x, or 10x, markup. By my last estimate, AWS marks up their egress costs by roughly 200-300x their costs! This is based on my estimates of what it would take to buy the network transit and routers/switches you’d need to egress a handful of Gbps. I’m sure this is an intentional lockin strategy on their part. That said, I have heard rumors of quite deep discounts on the network if you spend enough $$$. We’re talking 3 digits million multi-year commits to get the really good discounts.

My final point, and a major downside of cloud deployments, combined with a Service Ownership / DevOps model, is you can see your cloud costs grow to insane levels due to simple waste. Many engineering teams just don’t think about the costs. The Cloud makes lots of things seem “free” from a friction standpoint. So it’s very very easy to have a ton of resources running, racking up the bill. And then a lot of work to claw that back. You either need a set of gatekeepers, which I don’t love, because that ends up looking like an Ops team. Or you have to build a team to build cost visibility and attribution.

On the physical infra side, people are forced to plan, forced to come ask for servers. And when the next set of racks aren’t arriving for 6 weeks, they have to get creative and find ways to squeeze more performance out of their existing applications. This can lead to more efficient use of infra. In the cloud world, just turn up more instances, and move on. The bill doesn’t come until next month.

Lots of other thoughts in this area, but this got long already.

As an aside, for my personal projects, I mostly do OVH dedicated servers. Cheap and they work well. Though their management console leaves much to be desired.

I led that team for a couple of years. A couple of the strongest and most prolific engineers I’ve ever worked with. Who comes back from a 3 day weekend with a 5k line diff to review, that adds dramatic improvements to the core system? Well, I saw that happen more than once while working with that team. (No we never asked them to work through the weekend)

And San Francisco also legally limits the square footage of office space that can be built. Prop M passed in 1986 by the voters. Thank Sue Hestor for this.

This summer I bought a house in Tahoe (Carnelian Bay) and am now going through some mostly DIY renovations, with a little local contractor help here and there.

We also still have a place in San Francisco, but we're spending 70-80% of our time in Tahoe.

We spent the whole month of June here, and when we booked our place to stay, Placer county was not allowing short term rentals, so we booked for a full month to get over the threshold. While we were here, restaurants and other things opened back up.

About mid-June we started looking at houses. We made one offer for asking price, and got it accepted, right at the end of June. Overall I think we were fairly lucky. There were some stories of places going for substantially over asking price, which is rare for the area. But the largest impact was that housing supply is low, and houses moved really quickly.

It's typical for houses in this area to sit on the market for 3-6 months and sell for right around or slightly under asking price. This summer we saw a house go up on Zillow, on Wednesday, and we hemmed and hawed about it until Friday, then told our agent we wanted to tour it. The house was already off the market by that time. We saw plenty of places hit there market, and go into contract within 2-5 days.

We looked at several that we're real fixers. Like an 1100 square foot A-Frame that had a terrifying set of disclosures, probably needed at least a down to the studs renovation. All the electrical replaced, plumbing replaced, etc. Everything was DIY, not to code, etc. It sold in a handful of days for at least $500K, and certainly needed another $150K in work.

The place we did end up getting, we only got to see because our agent knew the seller and the selling agent. We got to tour it before it hit the multiple listing service and we had an offer in before it actually went on the market. It did hit the MLS for a few hours, but the buyer ended up wanting an easy sale, so accepted or asking price offer.

This kind of quick sale is typical for the Bay Area, but not at all for Tahoe.

I'm not keeping as close an eye on the market now, but overall asking prices are up, and people are still buying quickly. Also all the local contractors are very busy.

The other issue (and we're part of this problem) is that a lot of former rental houses are being put on the market, so locals are having quite a hard time finding a place to rent, and having to end up in places like Reno, which is roughly a 45-60 minute drive away. SFGate has an article about this exact issue here: https://www.sfgate.com/renotahoe/article/Bay-Area-transplant...

The other anecdote is that up until a couple of weeks ago it was still quite busy in town. People working at the local grocery store were commenting that it's the busiest shoulder season they've ever seen. Typically this time of year, this place is nearly a ghost town.

Both me and my partner have been working remotely since the start of March and expect to continue at least well into next year. Perhaps permanently.

And try to build higher than 4 stories in already built up places like San Francisco. Local groups will do everything to stop you. The heart breaking thing is Bay Area environmental groups like the Sierra Club are aligned with the anti-development crowd. They actively fight infill, but won’t acknowledge or see that encourages sprawl.

If I remember correctly, I got hundreds of pages of disclosures when I bought my condo in San Francisco.

Recently bought in Tahoe and less disclosures than San Francisco, but certainly quite clear about fire danger.

Gnome and KDE are in the ports tree and kept very up to date. Everything there basically works. Most supported hardware just works out of the box with very little tinkering.

Of course there is less hardware supported than Linux and support can lag a bit, since there are few people writing drivers. But modern Thinkpads work quite well.

All that said, I have found Gnome to feel a little slow on OpenBSD compared to running on on the same hardware on Linux. So on OpenBSD I mostly use dwm, which feels super fast.

Generally most everything you want can be found in the ports tree.

I still kind of miss the CD releases. I bought every release from 2.3 to somewhere in the 4.x series, and had them all in a closet until a few years ago. Unfortunately I got rid of them in a fit of cleaning.

I regret it now. Since then I’ve gotten back into OpenBSD as an occasional user on a random server here and there, and a Thinkpad X1 Carbon Gen 6. I bought it specifically for OpenBSD and it works great.

Those old CD sets were cool. Many included extra art work, song lyrics, install instructions and would boot and install for several architectures.

These days I mostly just run current snapshots, so a release has little impact for me personally. But it’s always fun to see OpenBSD still moving forward after 25 years.

I badly want this, but with Topre keys. I have a couple of Topre FC660Cs and absolutely love the feel, durability and quality of the keyboard. A split would be really nice though.

In many cases, like say the intel WiFi drivers on Linux, the driver is written by Intel employees.

The BSDs don’t have the critical mass to get this vendor support. For example, much of the intel WiFi work on OpenBSD is done by one person, Stephan Sperling, on a volunteer basis.

I only mention intel in the above examples because that’s what’s in the Thinkpad I had that didn’t have support yet. So I looked into the driver situation on Linux, FreeBSD and OpenBSD.

This sounds about right to me. The last time I was doing that kind of work it was at slightly large scale than that, and it always modeled out cheaper to stay with our on gear, in colocation space.

Yeah, this is really where they get you.

For context, if you were to buy 10Gbps of dedicated internet transit, he.net is currently advertising that for $900/month.

If we convert that to GB per month it's 3,240,000GB, so we can calculate what AWS would charge based on list prices.

Using their pricing calculator: https://calculator.aws/#/createCalculator

Outbound from Cloud Front or US West (Oregon) to the Internet:

$165,891.11

That's a 184 x increase in price!

So yeah, you have to buy networking gear and other stuff, but you can get quite a bit of gear for $165K/month. Now you don't really want to run that 10Gbps link flat out like that, but you get the point.

The AWS markup on bandwidth costs is absolutely insane.

Pro tip: if you have a large enough cloud provider spend, you can negotiate the bandwidth prices down quite a lot, given their markup, they have some room to move.

Yeah, this is correct. You usually buy these in minimum sizes of about 1MW.

Yeah, you can see the Direct Connect locations here: https://aws.amazon.com/directconnect/features/

Those mean AWS has networking gear in those locations. You order a cross connect and plug into one of their switches.

When you walk around in one of those places, you also typically see racks of AWS gear in a smallish cage with lots of hard drives. Typically a CDN pop.

When they say "running a datacenter" they almost certainly mean "buying servers to put into rented colocation space".

Just about anyone who has significant network connectivity has a footprint in an Equinix datacenter. In the Bay Area you want to be in Equinix SV1 or SV5, at 11, and 9 Great Oaks, San Jose.

If you're there, you can order a cross connect to basically any telco you can imagine, and any other large company. You can also get on the Equinix exchange and connect to many more.

But, Equinix charges you a huge premium for this, typically 2 - 3x other providers for space and power. Also they charge about $300 per month per cross connect.

So your network backbone tends to have a POP here, and maybe you put some CDN nodes here, but you don't build out significant compute. It's too expensive.

On the cheaper, but still highish quality end you have companies like CoreSite, and I'm pretty sure AWS has an entire building leased out at the CoreSite SantaClara campus for portions of us-west-1. (Pretty sure because people are always cagey about this kind of thing.)

I also know that Oracle cloud has been well know for taking lots of retail and wholesale datacenter space from the likes of CoreSite, and Digital Reality Trust, because it was faster to get to market. This is compared to purpose build datacenters, which is what the larger players typically do.

In the case of AWS, I know they generally do a leaseback, where they contract with another company who owns the building shell, and then AWS brings in all their own equipment.

But all these players are also going to have some footprint in various retail datacenters like Equinix and CoreSite for the connectivity, and some extra capacity.

Zoom is probably doing a mix of various colocation providers, and just getting the best deal / quality for the given local market they want to have a PoP in. Seems like they are also making Oracle Cloud part of that story.

I’m what would be considered a major donor to the YIMBY effort and I’ve donated in the low 5 figures.

It’s generally grass roots though I know some wealthy individuals have donated much more of the total funds.

The progressive camp of SF politics often tries to discredit the YIMBY movement by claiming AstroTurf. But it’s simply not true.

This was started by motivated individuals who started organizing and then went and started looking for donations.

The reality of SF housing politics is all the “local control” has given a lot of power to local groups that can hold a new development hostage. So developers have to essentially pay them off with “community benefit” to keep each group from delaying your project through years of appeals.

This sets up a system where a small number of developers have the relationships and know how to work the system to get their projects done. Limits local construction competition.

We also have local politicians (Aaron Peskin) who own significant amounts of rental properties, and find ways of opposing new housing supply. A policy they directly and personally benefit from.

Total not an expert in this area or really qualified to comment. But I think if they can get the known case load low enough, and continue aggressive testing and contact tracing, they may be able to lift the lockdown. Then aggressively isolate anyone who's come into contact with a known case.

If that doesn't work, restart the lockdown.

The big risk my be re-importing it back from other parts of the world who've been much less successful at containing this.

I worked at Go2Net when they acquired Dogpile. I was directly involved in migrating it’s systems to our data centers after the acquisition closed.

I don’t really remember the tech stack, but it was something like 10-15 web servers, probably pentium II or III.

One detail I do remember is the original setup had an Apache log rotation script that would try several times to restart Apache and if it didn’t succeed, it would reboot the server. This ran nightly...

Not sure what bug or issue they were working around. But I thought it was quite an aggressive solution!

Look at the track record of the SF Group of the Sierra Club. They consistent try to block dense urban development in the name is environmentalism. I went to their executive committee meetings for months, and even ran a failed campaign to unseat half the committee. They are very much part of the “preserve neighborhood character” and anti-density collation in SF politics.

Yes, and San Francisco is mostly zoned for no more than 4 stories in height. It would never be as cheap as Huston because it can't sprawl. But there is a ton of opportunity to build up.

The San Francisco height / bulk map is pretty enlightening. https://sfgov.org/sfplanningarchive/zoning-map-heightbulk-di...

All those light colored areas are max 40 feet.

This is one of the rare cases where I agree with the Trump administration. California does deserve all this criticism for years of policies that over-regulated housing, leading to a severe shortage.

And the California solution continues to be stricter tenant protections and more housing subsidies.

But the subsidies are nowhere near enough. This article suggests it would cost $12B for the Bay Area alone. https://www.cnbc.com/2019/04/10/cost-to-end-san-francisco-ba...

And when polled, 57% of Californians continue to blame the wrong causes: https://extras.mercurynews.com/blame/

Blaming developers, rather than the local governments that restrict supply.

I’ve been following these issues closely for years, and have been active in advocacy around them. I’m now convinced it won’t get substantially better without federal involvement.

I started my career at the peak of dot com, and remember one of my co-workers leaving. He’d been early at Wizards of the Coast. Someone else said he was “calling in rich”.

I have a radical idea that I’d like to see put into practice. Hopefully do it myself at some point.

We tell a story about risk and innovation that justifies founders and early employees getting massively more value (equity) than later employees.

And the amount of equity given out falls off a cliff as the company grows. So you have some people with ownership that is 10x, 100x or even 1000x larger than others. That leads to radically different outcomes.

What if we rethought equity rewards to substantially flatten those ratios? And maybe 20:1 or something was the max ratio.

Then employees truly are owners.

Certainly lots of issues with this half baked idea. But my overall point is that I think the typical equity pyramid is deeply unfair. It could be more equitable.

This. The bulk of the returns and growth now happens pre-IPO. Then the price at IPO is disconnected from public market expectations. But you’re also at the top of the growth S curve so growth is decelerating.

Artificiality high because HRC and similar organizations stop new housing at every turn...

Personally I use DigitalOcean for this sort of thing. But this is why AWS has Lightsail https://aws.amazon.com/lightsail/ to service that use case and then let people upgrade later to full AWS.

This. The major reason why I live in SOMA, and put up with all the significant downsides of that neighborhood, is because it means my commute is very short. It also means I have significant job and networking opportunities within a 30 minute _walk_ from home.

When I was last looking for a new role, I didn't even consider anything outside of SF.

I had several friends and coworkers end up in this situation. Lucky for me, I started late enough that my options were underwater before my first year cliff and stayed that way...

If you’ve vested any, which is usually 25% after the first year, and then monthly for the next 3 years. You get to keep those shares if you can buy them at your strike price. If you get laid off, you may only have 90 days to exercise (buy) your vested options before you lose that right. Some companies more recently are extending that time.

In the dot com crash era it was common for your options to have a strike price higher than the stock price (under water) so no sense in exercising them, since they are worth negative.

The thing I realized about OpenBSD is it’s made by and for the developers. They are going to do what works for them and fits their priorities. Not what works for the widest possible audience.