HNHacker News
TopNewBestAskShowJobs

kelp

1,528 karma · joined March 27, 2011

SVP of Engineering at PlanetScale. Previously Confluent Cloud Infra. Segment led the $10m eng problem project. Square built the data center and neteng teams.

[ my public key: https://keybase.io/kelp; my proof: https://keybase.io/kelp/sigs/BnRbIM-2QS4jsMclFvHJr9J9q-HioxWHvteK6DhGR1c ]

submissionscomments
kelp··on CA grid: real-time status
No I got the joke. My in-laws happen to live near Ontario and I kept having to think of the province in Canada to remember the name at first.

But I also knew about it because of tracking info.

kelp··on The new Ford Mustang has a drift stick
Regarding Ford branding, I feel like they've done a similar thing with the Bronco Sport, which is basically a reskinned Escape as far as I can tell.

Feels like it dilutes the Bronco brand.

kelp··on CA grid: real-time status
Ontario CA happens to be a major shipping hub with tons of warehouses. So it often shows up in UPS or FedEx tracking if you order a lot of stuff online.
kelp··on 30TB Portable SSD Hits Walmart for $39 but Stay Away from It
Or BestBuy that doesn't have the marketplace problem.
kelp··on California Declares Grid Emergency
Yes, the state is basically run by Democrats, but there are clear factions within that.

For example the recent special election for California State Assembly 17th district seat (San Francisco) was between Matt Haney and David Campos.

Campos is very much in the progressive camp, and doesn't believe in things like "supply and demand".

Haney used to be in the progressive camp, but he's the kind of politician that adapts his views to what he thinks will get him elected. So over the past few years he's shifted more towards the moderate camp, and ran on a much more moderate platform.

So, what you end up having is a far left group, and then more of a center left.

Then at the local level you have stuff like Shasta county almost getting taken over by far right groups: https://www.latimes.com/california/story/2022-07-12/shasta-c...

kelp··on California Declares Grid Emergency
The issues have certainly gotten worse over the past 15 years. A lot of factors combined to get us to the current state, and some were set in motion long long ago.

Just some examples:

Prop 13[1] passed in 1978:

* requires a 2/3s majority in both chambers to raise tax rates * caps increases on property taxes to sub-inflation rates, incentivizes people to stay in their homes forever to keep their cheap rate * forces the state to rely more and more on income taxes, which are very unstable, so we see boom and bust years. CA will be coming up on a bust year now. * Shields home voters from any downside of their property value going up (higher tax rates you'd see in other states), so more incentive to NIMBY any new development in your neighborhood. * cities can get more tax revenue from commercial properties than residential so they are incentivized to build office buildings and not housing (see how well that's working out for San Francisco now)

Late 70s downzoning: * Across California many cities downzoned, severely limiting the height, density and types of housing that could be built. * I read the environmental impact report for the late 70s downzone of San Francisco and it specifically called out how this would increase housing costs unless the city significantly increased development in the industrial parts of the city. That didn't really happen, so here we are.

The proposition system in general * We vote on absolutely crazy things that I think should be left to legislators or bureaucrats. One thing that stood out to me as particularly weird was 2018 Prop 8[2] * This history of this system was to limit the power of the rail roads who'd bought many of the politicians, so it was a counter to a corrupt legislature. But then someone "innovated" and started paying people to collect signatures to get something on the ballot, and now it's used for all sorts of confusing and niche purposes.

In recent years we've been seeing actual progress on some of the housing related policy. I think organizations like CA YIMBY[3] get a ton of credit for that.

But I'm not sure what we'll do about Prop 13 or the proposition system in general. Prop 13 in general is kind of a 3rd rail in CA politics. But there has been some attempts to chip away at the edges that got close. 2020 Prop 15[4] came close to passing and would have addressed some of the issues in Prop 13 for commercial properties.

[1] https://en.wikipedia.org/wiki/1978_California_Proposition_13 [2] https://ballotpedia.org/California_Proposition_8,_Limits_on_... [3] https://cayimby.org [4] https://ballotpedia.org/California_Proposition_15,_Tax_on_Co...

kelp··on 30TB Portable SSD Hits Walmart for $39 but Stay Away from It
Because if the problem is bad enough, people will start looking for alternative places to buy things. It's like how people will avoid the bad neighborhoods in a city.
kelp··on Amazon, Verizon found using IPv4 240/4 addresses
The thing about IPv6 is it has significant switching costs but almost no benefit to end users. The exception, I guess, is network providers who are struggling with a shortage of v4 addresses.

We've been trying to get people to switch since the late 90s/early 00s. And it's only now making progress because IPv4 address have an actual cost which keeps increasing.

But I think IPv6 is huge example of second system syndrome. Instead of just solving the obvious and most important problem of IP address exhaustion, we piled on a whole bunch of other requirements, greatly expanding the scope, increasing switching cost, and dragging out the migration for literal decades. Here we are 20+ years later with only 40% adoption of IPv6...

And I can see the temptation to bundle in all those other changes, since this might be the big opportunity to get them out there. But seems like we could have come up with a simpler and easier to switch to alternative that would have solved the acute issue much faster.

Instead we're here muddling through.

kelp··on Using the same Arch Linux installation for a decade
I always love hearing these stories, especially because I'm too fickle to stick with something that long. I've mostly been a Mac user since maybe Puma or Jaguar, but I disto hop every few years. Spent a good while on Debian, Ubuntu, Arch, FreeBSD and OpenBSD. Still have an OpenBSD Thinkpad x1 Gen 6 around somewhere. That OpenBSD install could age like this, but I fat fingered a dd command and nuked the system disk in mid 2020, so it got a fresh install then.

Had various Arch installs on desktop and Thinkpad that I used as my day to day for while, but I always just ended up back on my Mac. And I get a new Mac every few years.

Any Linux I've had always ends up getting torn down at some point. I get a new computer and do a fresh install.

kelp··on DigitalOcean Region Information
This actually surprises me quite a bit.

There are a lot of very expensive facilities on this list. Ones that I would not want to put much equipment in if I was at the scale I presume Digital Ocean is at.

But maybe it's a factor of organic growth and starting each region small, then having to grow into it. You can often get space for like 1/3rd the cost of Equinix but with a lot less services and maybe less quality. Equinix makes a very nice datacenter but you sure do pay for it.

If I were building a CDN or an edge compute product like Fly.io or Render, I'd certainly have a facility list that looked like this, because I'm charging a bit of a premium for excellent connectivity to population centers. Though I'd do some kind of managed hosting first until I had the scale to even get into collocation.

But for a bulk cloud provider I'd try to only use these facilities for things like network POPs, DNS services, CDN, etc. Then lease fiber to another facility that was a lot cheaper for the bulk of my compute. But that does take a certain threshold of scale to get there, and a longish planning horizon to make it happen unless you're growing at insane rates.

kelp··on DigitalOcean Region Information
Others have already mentioned that the connectivity in these places is really good, and that's often why you go there.

The other reason is you have employees there who can go fix things when they break. In the early days of Square we had everything in a single datacenter in Mountain View and our office was in San Francisco. There were times when something broke in a bad enough way that someone had to drive to the datacenter. It was a while before we built the scale and redundancy to put a facility on the east cost, or in places where we didn't often go.

But you're right, per square foot it's often quite expensive.

At a certain scale you still buy space in the city for connectivity, but then you buy IP transit, or even dark fiber, to connect to a cheaper and larger facility. That could be in the same region, or it might be out in some farmland with cheap power and good tax incentives.

We've seen lots of datacenters built in Washington and Oregon because the electricity is famously cheap hydroelectric. Like $0.02 - $0.03 per kWh. Last I looked a few (5-7?) years ago, industrial power in the Bay Area was closer to $0.10 per kWh and in Ashburn VA it was $0.08.

kelp··on DigitalOcean Region Information
Being inside those old repurposed telecom buildings can be WILD. I was in one that had overhead bundles of ethernet and fiber that had to be multiple feet thick, with the ladder rack bowing down under the weight. Elevator shafts were repurposed for running wires and fiber.

And then the building Google owns in Manhattan: https://en.wikipedia.org/wiki/111_Eighth_Avenue

That had a bunch of carrier hotels in it, I'm not sure if Google managed to kick them all out yet, since many had long term leases. But the building has a bunch of truck elevators in it... literally can load a truck up into the building. And a ton of fiber connectivity's which is why the carrier hotels were in there.

kelp··on DigitalOcean Region Information
TL;DR it depends on the use case, but I'd expect everyone in the top 5-8 cloud providers to own and operate at least some of their own facilities. But everyone is going to have some space in a collocation facility.

There are various degrees of "operate your own datacenter" in the datacenter market.

A lot of facilities on this list, like Equinix, are retail datacenters. They have things like 24/7 "smart hands" who you can call to go repair a server, replace or move a networking cable, things like that. These are the most full service options. They also offer various network connectivity products, and if you are willing to pay for it, you could go as far as ship them servers or a rack and have them do much of the install for you. These are the most expensive options, but, at least in the case of Equinix they also typically have the best network connectivity. This is your typical collocation, or even carrier hotel.

So if you go into an Equinix facility in a lot of markets you will see everyone with racks or cages full of equipment in there. Though the biggest players are mostly going to use them for network POPs and maybe something like a CDN node. So you might see Amazon CloudFront equipment in there, but not much else.

Often the thing to do is put your public internet connectivity, peering, and perhaps backbone links in a well connected place like this, but the bulk of your compute is in a much larger and cheaper facility elsewhere.

There are also retail collocation facilities that aren't super well connected (when I say well connected I mean you can get access to 10s or 100s of providers just by being in that facility) and those are a lot cheaper than Equinix, and often full service. They tend to cater to smaller customers. Though I've seen Akamai have a ton of racks in some surprisingly crappy faculties.

Then DRT (Digital Reality Trust) operates more in the wholesale datacenter market. They can rent cages for multiple racks, but are even more used to renting out whole datahalls, so you have dedicated space. In a lot of areas they can't really offer things like 24/7 smart hands. They aren't going to handhold you nearly as much, and space and power tends to be a lot cheaper, but your minimum buy is going to be a lot larger. 20-100 racks maybe?

CoreSite kind of straddles the line between Retail and Wholesale. They offer a lot of full service stuff, but can also rent you out a whole data hall or even building at a campus if you need it. No one would ever confirm this, but I'm pretty sure one of the big 3 cloud providers has a big chunk of, the CoreSite Silicon Valley campus. If you search Google Maps for core site silicon valley you'll see a cluster of about 5 buildings, all labeled. One building right there is suspiciously unlabeled, but if you do the the Satellite view, it's clearly a datacenter.

Take it another step, and you have companies like Sabey who offer a datacenter shell. Basically a building with power, generators, etc, but you do everything else, or they customize it to your specs and help operate it. Yet again cheaper per unit of compute you are doing, but a lot high minimum spend to make sense. You'll probably be measuring things in MW (megawatts) before you are looking at something like this. Sabey also sells to retail datacenters. I spent far too much time at the Sabey Tukilla campus south of Seattle from 2009 - 2011 in an Internap facility. I had to go through Sabey security to get into the building, and then Internap security to get into their space. Most everything inside that space was taken care of by Internap (now seemingly branded INAP).

Then you have build and operate your own datacenter, which also has various degrees. I don't know how it works now, but I know a few years back AWS had (has) a TON of buildings in the Ashburn Virginia area that are us-east-1. I was told by someone who worked in datacenter real estate that they typically would buy the land and get a datacenter shell built, then sell it to another operator and lease it back. Then bring in all their own power and compute.

I also heard that a few years ago when Oracle was building out their cloud, they were just buying up all the retail datacenter space they could so they could get to market faster. They were paying an arm and a leg for it, but that was the fastest way to get online.

So the economics of all this depend on what you are doing, why you have the space, and how much actual equipment you need in there. At smaller scales, say a CDN, or just networking equipment, it often makes the most sense to just lease retail space. At larger and larger scales it makes more and more sense to in-house more of it.

I've never worked at one of the large cloud providers, but I'd expect they have some smaller regions that are retail or wholesale datacenters operated by another provider. But the large regions are mostly owned and operated.

kelp··on FTC takes action to stop Opendoor from cheating potential sellers
Yeah, I understand the politics of it, and you're right that it's incredibly politically challenging thing to change.

But it's not a fundamental property of how markets for housing have to work. Housing doesn't need to be an appreciating asset for building housing to be a viable business to be in.

kelp··on FTC takes action to stop Opendoor from cheating potential sellers
Companies make cars, people buy them, everyone knows that a car is a depreciating asset (except for current used car shortages, and certain collectible classics) but people still buy them.

This is because they are very useful.

A house or an apartment is also quite useful without being an asset that appreciates in value. The whole rental market is based on this concept.

I don't see why housing needs to increase in value to incentives people to build it.

And in Japan this is exactly the case, it's not typical for housing to increase in value in Japan.

kelp··on VCs are scared when they should be greedy
From the linked article: "Critically, the venture market at the time was tiny relative to today’s ecosystem"

This isn't quite true. At least for US VC investment in dollars. It peaked at $66 billion in 2000, and didn't surpass that amount until 2018, according to these charts:

https://pitchbook.infogram.com/6-vm-charts-1h8n6m3klxngj4x

https://www.statista.com/chart/11443/venture-capital-activit...

And if you adjust for inflation, that year 2000 $66B is $103.86B in 2021 dollars, and the 2nd chart shows 2021 getting to $128B.

Now the two different data sources do have somewhat different numbers for each comparable year. I couldn't find a comparison that covered enough years to show the difference. But I think it's pretty clear that the dot com era was a spectacularly fast increase in VC funding. And the more recent years were slower growth, but did end up getting to slightly higher numbers, if you adjust for inflation.

kelp··on Blue Zones, where people reach age 100 at 10 times greater rates
So what’s the cause of high cholesterol and how do you lower it?
kelp··on Coinbase lays off around 1,100 employees
I'm actually in this situation on my property taxes.

In San Francisco there is a 2.5 month window to appeal your property tax assessment. I did that for 2020, and was successful (the hearing was like a year+ later) and they gave me a refund.

I failed to do it for 2021, because I was under the mistaken assumption that they'd they'd also adjust my 2021 assessment after my appeal. They didn't. I guess that's on me?

I just sold that property for a loss and 15% less than my tax assessment, and I'm well past the appeal period for 2021/2022 sales tax year. So I think I'm out of luck. I just over-paid by 15%.

kelp··on Shimano Forces Hammerhead to Remove All Di2 Related Functionality From Karoo
Oh I forgot, I also have one of those. So many computers on this bike.
kelp··on Shimano Forces Hammerhead to Remove All Di2 Related Functionality From Karoo
Yep and you can also set it to just use one shift lever for up and down shifting. It will shift front and rear as appropriate to get you a higher or lower gear. In practice this means sometimes it shifts the front and then moves the back up or down 2 or so gears.
kelp··on Shimano Forces Hammerhead to Remove All Di2 Related Functionality From Karoo
I have a 2020 Canyon Endurace with Shimanio Di2. I also added Garmin Vector pedals with built in power meters, and a Garmin edge bike computer.

I think all in my bike has around 11 devices on it with updatable firmware… And I might even be missing a few.

Both pedals, front and rear derailers, Di2 battery, Di2 Bluetooth adapter, Di2 Hub, both shift levers, Garmin computer, front wheel speed sensor.

And I often wear a Garmin heart rate strap, so that’s another.

kelp··on Ford CEO says EVs will be sold 100% online with nonnegotiable price
I bought my Volvo C40 EV this way, though it wasn't exactly seamless and the overall price I ended up paying wasn't as clear as it could have been, though it was all discounts in the end.

I put in the order through Volvo online, and they confirmed the build sometime the middle of last year.

Volvo had an online order tracker, that shows progress from order acceptance to the car being in production, etc. But it was never actually updated. I was a little worried the car wasn't going to show up before my existing lease was up in June. So I contacted Volvo customer support in early March and they surprised me by telling me the car was going to be delivered to the dealer the next week. I had heard nothing from anyone in months, so that was a big surprise.

I called the dealer, who I'd never talked to until that point, and they confirmed the car was coming in the next week and setup an appointment to pick it up. Because used car prices are crazy they could also buy out my leased ICE Volvo and give me a bit of money back for it.

So the price I paid (another lease) ended up including the new C40 at MSRP, minus trade-in, minus CA EV credit, minus Federal EV credit, and minus a Volvo loyalty discount.

It wall worked out pretty well, but would have been nice if their order status page actually showed order status. I imagine it was a manual process to update it and somehow that wasn't getting done. I heard from other they had the same experience.

kelp··on Starlink for RVs
Not here in my experience. My gear is is all on UPS. It will all be up, but the second the power goes out, so does Spectrum.
kelp··on Starlink for RVs
I bought a Starlink for my house in Tahoe to hopefully use as backup internet for when Spectrum goes down. They go down every time the power goes out, so even with backup power, I'd have no internet. Starlink would have solved that.

Unfortunately it's quite expensive for a backup internet plan. Being able to cancel and activate it at will with this new RV plan would help a lot with that. And while Starlink speeds are nice for what it is, I'd prefer to use my 1Gbps cable modem most of the time.

Sadly for me, I tried putting the Starlink dish in several spots on my roof and deck, but I just have too many trees around my house. 100 foot tall jeffry pines all around the house. Starlink was always reporting about 12% obstructed, so it expected a drop out every 45 seconds or so. It was usable, but packet loss here and there.

I don't think that was going to work well for any kind of video calls, etc.

So I ended up returning the Starlink dish. It really needs too much view of the sky for heavily wooded areas. They aren't kidding when they say that.

My fall back plan is to use a 5G/LTE hotspot with an outdoor antenna to make up for my usable but fairly week cell service in the area, and use that for my backup internet.

kelp··on OVHcloud fire: SBG2 data center had no extinguisher, no power cut-out
This does seem crazy to me. I've probably toured 30-40 datacenter facilities in my career, and they ALWAYS have a big red EPO (Emergency Power Off) button at the major exits to each datahall. I've been in plenty of facilities all over the US, Europe and Japan. (I've also had to deal with the fallout from outages caused by someone accidentally pushing that button. I think they thought it would open the door. Later on those buttons were always covered with a clear plastic housing)

Though never in France and everything I've spent time in was a a retail or wholesale provider, eg selling space to other companies. Not something owned and operated by a single company.

kelp··on How our free plan stays free
I struggled with a use case at first as an individual user, but now I'm using it in a few different places.

I have a Synology on my home network which I use for Time Machine backups among other things. My Mac has a Tailscale client and I can backup to my Synology from anywhere.

I have a number of random servers I keep for hobby stuff, a mix of hosted bare metal, VMs and VPS. None of them have SSH open to the internet. My access is all over Tailscale. It was super easy to setup, and now I never have to touch it. Occasionally I'll see that the Tailscale daemon was updated on some host.

If I were starting a company today, as soon as I had any resources that needed any kind of remote access for the team, I'd use Tailscale to provide that access.

kelp··on Things you notice when you quit the news (2016)
I don't think there is objective agreement that the Economists views are right-wing neoliberal capitalist.

AllSides rates them as "Leans left": https://www.allsides.com/news-source/economist

They've endorsed Democrats for US President since 2004.

Ad Fontes also rates them slightly left leaning: https://adfontesmedia.com/the-economist-bias-and-reliability...

kelp··on The Unreasonable Math of Type 1 Diabetes
This is a great idea, thank you! If I can get myself into shape for it by late June, I'm going to try doing the JMT this year. This tip may come in handy. Thanks.
kelp··on The Unreasonable Math of Type 1 Diabetes
I just looked at your profile and saw what your company is working on. Very interesting!

My wife and I were recently in a situation where I'd wondered why the hospital didn't have such a thing!

My wife has T1D and we just spent 3 days in the hospital for the birth of our daughter. The most scary and frustrating part of the whole process was my wife's diabetes management. She uses a Tandem T:Slim and a Dexcom G6, and generally has very tight control. But the hospital insisted on taking over during active labor, and we had a very tense conversation with a tactless perinatologist about it.

They wanted to put her on an (IIRC) intermediate acting insulin drip, and rely on finger pricks. We were very worried that they would basically be following a canned script and would end up yo-yoing her blood glucose during labor.

We ended up getting our OB to convince the completely tactless perinatologist that we'd define "active labor" as actually pushing. So we delayed getting her off her pump/CGM and self management for quite a while. And by then we'd also fortunately convinced our nurse who was responsible for the insulin that we knew what we were doing. So she consulted with my wife (literally between contractions and pushing) on any insulin delivery. They adjusted the plan in real time based on what my wife said they should do. Had they just followed the script/protocol the hospital had, we probably would have seen them give my wife way too much insulin.

Then for the rest of our stay, the hospital insisted on checking her blood glucose with finger pricks, though she was self treating with her pump. It really felt like it was for their benefit and not hers.

It was really clear that literally every person we interacted with (except maybe the tactless perinatologist) knew less about T1D than we did. They are much more setup for poorly managed gestational diabetes.

Watching the hospital try to manage her T1D made us feel like we were jumping back in time a decade or two. And this was one of the big, nice, (and expensive) hospitals in San Francisco.

If she would have had a c-section or some other situation where she wasn't able to help direct her diabetes management, I'm kind of afraid of what could have happened.

It really made me wonder why there wasn't an artificial pancreas type system for hospitals.

Now I see that someone is actually working on that!

kelp··on The Unreasonable Math of Type 1 Diabetes
This kind of exists, and here is one such system:

https://www.tandemdiabetes.com/products/t-slim-x2-insulin-pu...

The pump delivers insulin and gets blood glucose readings from a CGM (continuous glucose monitor) over bluetooth. Then the pump can make decisions such as stopping insulin delivery if you're going to go low, doing small correction boluses, and adjusting basal rate.

However it's far from perfect and still needs a lot of management. My wife has T1D and was recently pregnant, which really raises the bar on how tightly you need to control your diabetes. She ended up mostly micromanaging it all herself rather than trusting Control IQ to do the right thing.

The other issue is CGMs are not that accurate, so these systems are making decisions based on not great data.

← PreviousPage 3 of 10Next →