kelp

Re AWS Linux support, ~free is about right. You do pay for it. At enterprise levels, you're paying a percent of your total bill for support. That can be significant.

https://aws.amazon.com/premiumsupport/pricing/

But with it you generally get very quick responses to issues, and if your bill is enough, you get a technical account manager who gets paged when you open a ticket. They handle many things, but I believe AWS Linux is in that list of many.

Having the Technical Account Manager is quite valuable.

We wrote our own thing because at the time people were not all doing that with Ansible.

Back when I lead Network Engineering at Square, we had a global network of production dataceters and offices. Nothing of the scale of Cloudflare, but we still have many hundreds of devices. They had all been built manually with copy and pasted configs. The trouble with this was it made changes very risky and terrifying because there could be subtle inconsistencies between sites, or huge differences. So it was always very challenging to reason about the impact of a given change.

Thanks to a ton of grit by the team, and the insistence of one engineer in particular we built a config management system and started tracking the total percent of our global network config that was managed by our config management system.

That metric was regularly presented at the VP level to hold us accountable to getting the percentage to 100.

It was months and months of boring work to remove inconsistencies and templatize configs. But in the end, I believe it resulted in a much more reliable and ultimately safer network to operate. I'm also happy that my management chain saw the value in this work.

I'm quite proud of the work the team did.

Some side benefits were that once we started going through audits like SOC2, we had a really good story to tell about how we reviewed and pushed changes to production.

Thanks for pointing this out. Seems right!

Mostly it's business growth stalling with no clear plan to fix it.

If you don't have visibility into that, there are other warnings signs, like various spending cuts. No straight answers to questions about that stuff.

Not providing visibility into the business is also a warning sign, in my opinion.

I've also left companies when they started making pretty weird business bets that I dind't think would pan out.

This is purely my own speculation informed by some Clayton Christiansen, but it's possible that someone at Ford sees Ford as a transportation company, not a car company. So the job to be done is moving people and things around. Ford also sponsors the Ford GoBikes that you see all over San Francisco. This could be seen as competing with the Ford the car company. Or it's just part of the portfolio of products from Ford the transportation company.

That kind of thinking could help them mitigate disruptive pressure, especially in markets like SF where driving is pretty awful, but so is transit.

Perhaps also luck, but I’ve managed to have a spidy sense for when things are going the wrong way, and got out / moved on before it got that bad.

This is about my career. 8 startups, earlish employee at 6. 1 mediocre IPO, and 1 very successful IPO.

The rest basically failures or zombies that made me no extra money.

Probably a little more lucky than most.

Oh you’re right! I’d missed the launch of t2 unlimited.

Looks like with t3 they’ve swapped the defaults. T3 now defaults to unlimited, t2 it’s an option.

I'm by no means an expert in this, and maybe it's a bit obvious, but hadn't seen this mentioned yet.

I think as we run out of gains to be had from process size reductions, the next frontier for cloud providers is in custom silicon for specific workloads. First we saw GPUs move to the cloud, then Google announced their TPUs.

Behind the scenes, Amazon's acquisition of Annapurna Labs has been paying off with their Nitro (http://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtuali...) ASIC, nearly eliminating the virtualization tax, and providing a ton of impressive network capabilities and performance gains.

The Graviton, which I believe also came from the Annapurna team, is likely just the start. Though a general purpose CPU, it's getting AWS started on custom silicon. AWS seems all about providing their customers with a million different options to micro-optimize things. I think the next step will be an expanding portfolio of hardware for specific workloads.

The more scale AWS has, the more it makes sense to cater to somewhat niche needs. Their scale will enable customization of hardware to serve many different workloads and is going to be yet another of Amazon's long-term competitive advantages.

I think that will show up in two ways. Hardware narrowly focused on certain workloads, like Google's TPUs that show really high performance, and general purpose CPUs like these Gravitons that are more cost efficent for some workloads.

I see echoes of Apple's acquisition of P.A. Semi that lead to the development of the A series CPUs. My iPhone XS beats my MacBook (early 2016) on multi-core Geekbench by 37%. (And on single core, it's only 10% slower than a 2018 MacBook Pro 15.)

If Amazon is able to have similar success in custom silicon, this will be a big deal.

I think early next year we'll test the a1 instances for some of our stateless work loads and see what the price/performance really looks like.

It does make me worry that this sort of thing will cement the dominance of large cloud providers, and we'll be left with only a handful (3?) of real competitors.

FYI, t3 instances have and option to either be throttld when you exceed your burst credit or just pay more for burst.

This can't be overstated for a cloud provider, especially one like AWS that wants to run everything for everyone. ARM has good OS and compiler support, and if you want people to move some workloads off x86 to another architecture, you're gonna want that migration to be as painless as possible.

From the perspective of capabilities, I think you’re right, the market is nearly fully served. From usability, I think we still have a long ways to go.

Not sure if you're making a joke or unaware of https://www.gnu.org/software/gawk/ :)

I keep meaning to write up a longer form version of this idea that I've been kicking around to solve this.

This issue is often talked about as an unsolvable problem, but I feel like we already have the technology to deal with it.

Lots of devices, such as an iPhone already have a secure enclave that can be used for identity. Why not use this to sign videos and images for authentication purposes?

Then certain devices can produce verified, undoctored images and videos. Or even allow some amount of limited tweaking, and editing while attesting to the original.

This could be embedded into the file produced, like exif data, and read by services like Facebook and Twitter to badge the image/video as verified and undoctored. This could improve the amature case significantly.

Professional photography and video equipment could have similar capabilities so originals could be produced and proven to be genuine.

I guess this idea still depends on the hardware devices remaining secure, which is far from trivial, but it does increase the cost of producing a fake. And to keep producing really good fakes, you have to have a zero day to fake a genuine signature, or it becomes know that any media produced with that device version is suspect.

If such a system became widly adopted, at the very least, savvy users, and the media would be suspect of anything not signed. And perhaps the general public would also learn to be skeptical, particularly if the issue of fakes became more widespread.

What if because of the opportunities available, San Francisco creates more rich people. I know many tech workers who start out in small apartments with many roommates and then end up quite wealthy.

I think the issues with the comment is the implication that the wealth creation is the problem. That framing leads to solutions that tend towards reducing the wealth creation. That seems a bit like cutting off our nose to spite our face.

We want society to be more wealthy, and the tech industry has been a huge creator of wealth. The other thing about the tech industry is it's fairly unique in how widely it distributes that wealth. What other industry gives stock options to low-level employees? Sure tech founders get a disproportionate share, but look at a different industry, and it would be rare to see any ownership extended beyond higher level employees, but in tech, stock compensation is ubiquitous.

(this may be due to a labor shortage, and yes you can point at things like Amazon's warehouse workers as a counter example)

The trouble is now we have quickly raising housing costs, and a large portion of the population who's quite wealthy, living next to many who are not doing nearly as well. Crime and general unhappiness are not necessarily linked to poverty; it's highly linked to inequality. This is likely why we have so much property crime in San Francisco. Income inequality. When you see the person next to you is doing much better, you get mad.

Frame this so it's not about limiting wealth creation, but about limiting income inequality.

The solution is to means test property taxes. To keep lower-income residents in their homes, we discount their property tax from the standard rate.

If you're wealthy and can afford to pay a market rate property tax, then you pay it.

Isn't it quite an oversimplification to say that the homelessness issues in SF are due to stupid politicians? Do other cities somehow have better smarter politicians?

I think it's more likely that a series of cultural, historical, systemic and environmental issues have lead to the current situation, and that our politicians are generally average in competence and intelligence compared to other similarly sized cities, but are dealing with some particularly challenging issues.

Some possible factors:

Cultural: Other cities, such as NYC, house many people in large shelters, on cots, outside of the city center. SF wants to put people in a higher standard of living than a room full of cots, so we can only afford to house about half the people who need it. (Much of that $250M/year we spend goes to housing people) It's hard for the BART police to kick people out of the stations because homeless advocacy organizations complain about abuse. So the problem stays visibile. Places like Salt Lake City also see the Mormon church active in improving these problems. I don't think there is any such non-govermental organization working at such scale in SF. Generally we think it's the goverments problem to solve, other places think it's a community problem, and take action. We seem to tolerate people visibily using drugs, leaving needles around. More conservative cities would not let that happen a block from city hall.

Historical: Most of SF's services, such as shelters, needle exchange, healthcare, etc, are downtown, this makes the problem very visible. NYC pushes this out of Manhattan and has for years. Neighborhood groups in SF usually protest adding a navigation center to their neghborhood. Imagine if we started putting shelters and navigation centers in the richmond?

Systemic: This could be an essay alone. Rising housing costs, due to downzoning, prop13, discrtionary review, etc etc, and then a job loss, and you can't make rent, you lose your apartment. Now you're out on the street. The city doesn't have enough shelters. And see previous mention of neghorhoods not wanting navigation centers or shelters near them. The city actually has to go in front of the planning comission, just like a private developer, to argue for a new navigation center. Not only do we have a system that makes it hard to build new high end housing, it's hard to build navgation centers, because of all the process involved.

Environmental: California cities have a much higher rate of unsheltered homelessness. NYC has a legal right to shelter. I do wonder if part of this has to do with climate. If NYC had the same number of people living unsheltered through the winter, they would have thousands of people freezing to death. NYC has 75K people experiencing homelessness at any given time. This creates the political will to house people.

So you're elected to the board of supervisors, or as the mayor, you have all this history, all these factors that are making your job of reducing homeless, very hard. What do you do? We've had many many bonds to increase funding to provide housing for people experiencing homelessness, but the electorate consistently votes them down. So your hands are also fairly tied with regards to money. You're left with small incremental imrovements over many years.

I can see why you'd be fed up with San Francisco for these reasons. I just don't think it's fair to blame it entirly on stupid politicians. All that said, I do agree with you that I'd rather see the time spent on more pressing issues than tech company cafeteria bans, and plastic bag bans. Part of this is because we elect people like Aaron Peskin who seem very interested in 1. having the ear of local small business groups, and passing protectionist laws. 2. Putting forward sensational laws that get a lot of media attention. Peskin and his ilk operate on the emotional appeal side of politics. Things that sound good as long as you don't think too hard about them. We'll find those types of politicans everywhere. We have to work to expose them for what they are.

I'd like to add that the Bay Area affordability problem is not some unsolvable situation that we have to just be resigned to accept. But it does require getting involved somehow. If you have a decent paying job, the easiest way is to start donating to organizations like SF YIMBY, and / or political campains of pro YIMBY candidates. People such as Sonja Trauss, Christine Johnson, who are both running for SF D6. David Chiu and Scott Wiener have had a great track record in the State Assembly and State Senate.

We got into this affordability situation by passing a series of laws at the state and local levels, over many years. We're never going to be as cheap as Akron OH, but we can slow the price growth, and maybe reduce it some by making it easier, and cheaper to build, by changing state and local laws to allow that.

Finally, local politicians are actually quite accessible, so your voice can be heard. And groups like SF YIMBY help to amplify that voice. Fundamentally, most politicians do what they think will get them votes, so it helps to find a group of like minded individuals.

Part of it is likely density of companies, talent, funding, advisors, etc. The whole ecosystem for starting a technology focused compnay is right in your backyard. In my case, I live in SOMA, and was recently looking for a new job. I talked to 20+ companies, all of them within a 30 minute walk of where I live.

I can easily head over to a friends company for lunch and learn about what they are working on. Or invite a former co-worker to our office for lunch to try to recruit them. All of this is maybe 15 minutes travel time in SF. I don't know how to quantify the value of this, but I don't see it being replicated anywhere else.

And then combine that with a very open culture, where people are happy to share details about what they are working on, it really helps to accelerate learning. I lived in Seattle from 1999 to 2009, and did not experience the same open culture there. Maybe it's improved since, but I was amazed at how much people would share when I moved to SF.

To add to this, I think it would be a worse sign for the company if they have a number of people in such a critical path, that they can never take significant time off. If you're doing your job well as a leader (and an organization), you have people who can fill in for you.

There is also this theory that it's caused a Hepatis outbreak: http://www.sandiegouniontribune.com/news/hepatitis-crisis/sd...

People who are homeless would often use plastic bags as a bathroom, and then throw it in the trash. No more plastic bags, so they can't do that...

Not sure what real evidence there is for it, but I always love an unintended consequences story.

I live in San Francisco, but I've spent this whole winter in Utah, and it's actually been quite wonderful.

This is true for the most part. NYC has more homeless, but they are kept out of Manhattan, and the supportive housing is in the outer boroughs. That said, I think NYC also does a better job of housing their homeless.

But, to your point about being too liberal, I've seen some of the most disgusting things ever in Powell BART station, and when the police try to kick people out, they get too many complaints and have to back off. So we all have to suffer.

I suggest lobbying your state and local representatives to allow fast and easy approval of building more high-density housing and to upzone your neighborhood. The reason California is so expensive is because the generations who already own homes have long asked to maintain the status quo with low height limits (most of San Francisco is 40 feet!) and density limits. This creates an artificial shortage. And those homeowners have somehow created a political coalition with lower income renters afraid of gentrification by offering them things like rent control and token amounts of subsidized housing. The thing is, if you do the math, it quickly becomes obvious that there is no world where we create enough subsidized housing to make a dent in the problem. And rent control works to keep people in their homes but at the expense of everyone else. Subsidized housing is a lottery, and always will be in California. We don't suggest buying lottery tickets as a solution for income inequality; we shouldn't for housing inequality either.

The only viable long-term solution is relax zoning laws, and make new construction approvals easier and faster.

SB-827 is a start in the right direction.

It takes about 3 months to pay off a $1200 - $1600 mining machine, nearly 4 if you have to pay for electricity. That all assumes the market stays in it's current state for that whole time. Highly unlikely.