What It Costs to Run Let's Encrypt | Hacker News Reader

What It Costs to Run Let's Encrypt

letsencrypt.org

680 pointscpeterso373 comments

To everyone complaining about the $250K (see edit, it's actually 200K) per person in salary (which seems to be everyone at the moment), a few things to keep in mind:

1) That includes benefits

2) They are in Silicon Valley and have to compete with everyone else in Silicon Valley

3) Given what they have accomplished with such a small team, these are probably high caliber people.

4) In SV there is a good 25% premium you pay for engineers with security expertise.

As someone who runs a business and has to hire talent, I can say that I'd consider myself lucky to get such a team for such a great price.

Edit: A bunch of people are saying "why not remote". I think they are in fact mostly remote, but they still have to compete with SV wages because really good people command those wages whether they are here or not.

Edit2: As has been pointed out multiple times, the blog post was not clear, and it is actually 200K per person, all in.

> I can say that I'd consider myself lucky to get such a team for such a great price

I am concerned that they might be being underpaid for doing the type of work that they are. Let's Encrypt is not only a great service, but an important one too. I assume they have a good stock plan to make up.

Since they're a project of the Linux Foundation, they most definitely don't have any sort of stock plan – since you can't have equity in nonprofits.

I don't want to put words in the parent commenter's mouth, but I think the implication was that they would need to make up for NOT having a stock plan.

Netflix level pay would seem about right to me.

So, double? $400k loaded? I'd buy that.

LF is more of an industry consortium than a nonprofit public interest charity.

I would hope they have high caliber programmers and ops people, considering the problem space. Those don't come cheap.

Can anyone comment on tax overhead for the company in SV? What net salary do 250k amount to? (net as in minus the tax for the company, not the employees income tax)

It depends a lot on corporate structure and actually HQ address for tax filling. Each city has their own business license rates and deductions. And some cities (like SF) have extra taxes per employee. Their website lists their HQ as SF, but it would be impossible to know from the outside what their exact tax situation would be.

I guess my point is though from a company perspective not much more than anywhere else. The biggest cost to a company here is competing on wages.

I've worked in finance for a medium-sized SF company and a small (~50 employee) startup in SV. Our standard assumption for 'all-in' employee cost was to add an additional 30% on top. For highly paid employees, this actually gets reduced a bit since FICA cuts off at $118k.

To get to $250k total outlay with an admittedly outdated spreadsheet, the salary would have to be $200k (benefit load of only 26%). Of course this depends greatly on PTO, Health Care, Worker's Comp, etc.

There isn't as much as people bitch about, generally speaking you have a few items:

- FICA payroll tax - this is US wide, so not really more expensive in CA. - CA state payroll tax, this isn't that expensive, just a few percent. - Workers comp, could be cheaper in other states, but coders/office workers have cheap workers comp. - Health insurance, maybe a bit more expensive in CA

Basically hiring people isn't expensive because of the taxes, but because of benefits and just generally having a higher income rate. In my experience it was about 10%, and if you add health care another 10-15% so about 25% total overhead.

Also office space: 100 - 200 sq.ft. per employee * $75/sq.ft. in SF -> $7k5 - $15k per employee.

That's like for the MOST expensive office space in SF though. Also that's PER YEAR, not per month.

I once rented out office space in SOMA, and it was more like $18 sqft/year. It was cheap and in a shitty place, but still SOMA.

That's average price according to http://fortune.com/2016/01/11/san-francisco-office-rent/

I'm in Silicon Valley and I would love a $250k salary.

But I also didn't graduate college and incur the immigrant tax. ¯\_(ツ)_/¯

What I'm saying is, not everyone gets $250k and the MEDIAN HOUSEHOLD INCOME in San Francisco is $90k. If you're not making that much, don't sweat it, you're likely way above median either way.

Edit: to clarify further. As an Engineer with a reasonably-employed girlfriend, I still get to live in SoMa, and I'm not complaining. But statistically speaking my household is an outlier. $250k/year for a household puts you in the top 1.5% in the country[1]. That's probably top 5%-ish for San Francisco. That means that for every 5 households that make this much, there are 95 that don't.

[1] https://en.wikipedia.org/wiki/Affluence_in_the_United_States

They're not actually making $250k each. Keep in mind that this includes employer taxes and benefits, which add up very quickly.

OK, so $200k in salary then. Still stupendously good. WOW!

Maybe there _IS_ something to all these "I know this guy who's brother's roommate makes $300K at Google" threads. It just seems so unbelievable.

I'm surprised that there remains so much disbelief regarding these figures. There are a plenty of people at large software companies in Silicon Valley who are paid well north of $300k.

https://www.glassdoor.com/Salary/Google-Senior-Staff-Softwar... https://www.glassdoor.com/Salary/Dropbox-Engineer-Salaries-E... https://www.glassdoor.com/Salary/Facebook-Staff-Software-Eng...

Google has thousands of engineers. How many "senior staff software engineers"?

"This estimate is based upon 18 Google Senior Staff Software Engineer salary reports"

"This estimate is based upon 1 Dropbox Engineer salary report"

"This estimate is based upon 1 Facebook Staff Software Engineer salary report"

So we're looking at salaries of 20 engineers. There are how many thousand engineers in the Bay Area?

I don't think your data is statistically significant. I'm not saying these people don't exist. I'm saying they're not as common as these companies would like us to believe.

Yea, I always understood those Super Senior Staff Software Engineer roles at these top companies are for the rare "I invented the TCP in TCP/IP" people, not the more common "third average software engineer from the left".

This is an incorrect understanding. I personally know someone at Google who is nowhere near "I invented the TCP in TCP/IP" and they make >$300k (but not much more) with just a few years of experience.

From what I've been told, take the employee's salary and multiply by 2 to get the actual cost to the company.

To hire them, maybe. To employ them (after the cost of recruiting), it's more like 30% on top of salary.

To clarify, average $200k per person in staffing costs. Staffing costs > salary.

Yes, depending on the level of benefits offered and local taxes, you are looking at 20%+ cost on top of salary to employ someone.

As others have pointed out, with benefits, taxes, etc the actual salary is going to by way under that.

But I don't see why you're getting downvoted either. The income inequality in places like SV and Seattle/Redmond is insane. People working "normal jobs" struggle to get by and often have to commute further and live way outside the city just to survive.

It gets even more ass-backwards because public transport hubs are way more expensive to live near and, once again, only highly paid individuals can live near them (and therefore ditch their cars potentially saving even more money, where as the not so well off have a car requirement because .. American's transportation failure. And before you give me that "American is too spread out for rail" shit, Russia has three high speed rails and St Petersburg has the 2nd largest tram network in the world).

You're pointing to a very real problem. If there was plenty of housing and transport, those wages wouldn't put the strain they do on every one else. But there isn't and they do. Working remotely would actually be way better because at least they're not in the SV area.

And to rant a little more, a high end engineer that would cost $200k in Washington/California (maybe NYC too) would start at 50k euro in Berlin. They'd top out around 70k euro at the most. Even for really highly skilled positions, Germany tends to have way less income inequality and flats are also way more affordable for everyone.

> And to rant a little more, a high end engineer that would cost $200k in Washington/California (maybe NYC too) would start at 50k euro in Berlin. They'd top out around 70k euro at the most. Even for really highly skilled positions, Germany tends to have way less income inequality and flats are also way more affordable for everyone.

Which is why I moved to the States in the first place. I may have fallen from the top-ish of the pile to the bottom-ish. But the glass ceiling is waaaay higher.

Income data does not give you the full picture, especially in a city like SF. A lot of people have inherited wealth or trusts that provide moderate income (at least as reported to the IRS), so looking at income alone is insufficient for understanding wealth profiles.

My neighbor runs a small business and makes $60k/year. But he lives in a house he inherited worth over $3mn while paying taxes on the value it was assessed at in 1963.

This answers my perennial question of "How the hell do these people survive??"

Any idea where I could find wealth data? Would be super cool to research as I spend a lot of time thinking about stuff like this.

The 2M staffing budget in 2017 is for ten people, not eight. Was not clear enough in the blog post initially, apologies.

Actually it is for 8. Two of the people are being paid by another company. At least that's what I understood.

Heh, I'm pretty sure OP actually works for Let's Encrypt.

Lololo

It is ten (I made the budget), but I apologize for not making that more clear in the blog post.

We have eight employees now, and have budgeted for ten in 2017.

OK, thanks for the clarification. You guys are doing a really awesome thing but this post makes me worry about your longevity. Does Let's Encrypt have plans for other sources of revenue? Like selling support services?

More to the point, if anyone truly believes they're able to do it for cheaper... Do it, don't comment about it.

> Edit: A bunch of people are saying "why not remote". I think they are in fact mostly remote, but they still have to compete with SV wages because really good people command those wages whether they are here or not.

This doesn't make sense. In the case of remote workers it's the San Francisco workers that have to compete with global offer/demand because nobody cares how much you are paying for housing.

It doesn't make sense but it is true. If you're really good, someone will make you an offer for an SV salary, and now everyone needs to compete with that. Besides, it's only fair. Your value shouldn't be tied to where you live per se. It's true that people in some areas make less because they have fewer options in their area, but for a high caliber person, they have plenty of options whether they move their physical location or not.

I'm pretty sure we're going to see a trend as remote work becomes more popular where there will be a much smaller delta because of location of the employee.

Yes, exactly. Remote is incidentally a way to save some money on salaries, but mostly it is a great way access a much larger pool of talent.

> Your value shouldn't be tied to where you live per se.

If that was the case people wouldn't make $200k in SV, $75k in Omaha, and $9k in Phuket for the same job.

People who are near the top of the skill curve, and can present a business case for being paid like it, don't make $75K in Omaha for that job. (They may make $9K in Phuket, but there are additional factors on top of that.)

If I left Boston today, I'd still make what I make here. I have considered doing exactly that.

That really depends on very much more than your skill as an engineer, and the ability to "present a business case for being paid like it" simply does not apply to the majority of employment negotiation scenarios--even ones involving the very best of the best engineers. Most, in fact almost all employers have fixed salary schedules for their positions and they simply will not deviate from these as a rule, nor will they entertain negotiation around them.

> Most, in fact almost all employers have fixed salary schedules for their positions and they simply will not deviate from these as a rule, nor will they entertain negotiation around them.

Most employers will happily tell you that this is the case. That does not mean that it is so. If you rate it, those doors will, at many companies, open. But you have to rate doing something that is Not Normal, and you have to be able to communicate why you rate having the people dealing with your hiring stick their neck out and do something that is Not Normal. This is hard. Few people (not just engineers) can do that. But it can be done.

It means exactly that it is so for almost all cases. There are always going to be exceptions, but those are just that: exceptions. There's a reason they're called "exceptions." Edit: the people who set compensation are very rarely the people that decide whether to hire you. The latter may communicate the salary HR dictates to them, but they almost never have negotiation power within their own employers' structure.

I've worked at 20k+ employee companies where these are split into entire separate decision-making structures. Salaries are set by HR and approved far up the chain, and hiring decisions are made by people who do not share supervisors until you're a level below the CEO.

When you're hiring a "Programmer II" in a company like that, the salary is not going to change. You might be able to negotiate work-from-home a day or two a week, or an extra week of vacation or something, but you're simply not going to make more than the upper bound of that schedule, and being at the upper bound of that schedule will negatively impact raises until you get promoted.

The CEO or COO or CIO is not going to approve a $100k salary for a Programmer II when the band is $60-75k. It just won't happen.

On an extremely related note, there is zero ambition and very little technical skill at that company. The vast majority of folks with skill and ambition leave after 2-3 years.

It doesn't have to apply to the majority of employment negotiation scenarios. It just has to apply to enough of a particular class of worker to set the market price.

I would also suggest your understanding of how often exceptions are made is very unlikely to be based on good data. Employers have ways of hiding that they are paying somebody more than their traditional salary structure when they need to make an exception. And a very strong incentive to do that hiding.

Yes, and the same reasoning would apply to folks suggesting that employers routinely are open to negotiation. On the other hand we have a plethora of anecdotal data (e.g. through Glassdoor and other self reporting places) and BLS salary data by region and classification (https://www.bls.gov/oes/current/oessrcma.htm). The means and medians in these data are quite close. It'd be nice to see upper and lower extremes, modes and other information to gauge the actual distribution, but it strikes me as an extraordinary to claim successfully negotiating higher-than-average wages is typical.

Who's saying anything about typical?

I've never been a typical employee in my life. I have, however, been the highest-paid non-manager (or so I was told; this might have been an attempt to shame me into working more for free) in a hundred-person development group, well out of band, because I brought capabilities and skills to the table that they needed more than they needed to Maintain The Integrity Of The Salary Band.

So if your point wasn't that employers routinely deviate from their prescribed bands ("as a rule") why did you respond to my comment? What I see is an anonymous Internet Person bragging without context (for all I know your non-typical experience could have been with 99 developers who were seriously below average).

They won't routinely deviate from prescribed bands for your typical engineer. There are many, many engineers who are atypical, whether because of skill set or skill level; before I went into consulting I was able to command top-of-market salaries because I am pretty good--not the best, but pretty good--and actively went into areas of technology that others didn't. I was then able to communicate to potential employers why this mattered and how I would be able to help them, because I have enough of a business background to be able to make the case for myself. And while whatever "talent" is involved may be innate--I don't think it is, but whatever--the rest is eminently learnable.

And calling me "anonymous" is kind of funny.

If you see him as anonymous, you are definitely not trying very hard. Or at all.

I'll be seeing you at the next Anonymous Anonymous meeting, right, Anonymous?

You know it, A.

> Yes, and the same reasoning would apply to folks suggesting that employers routinely are open to negotiation.

No, it wouldn't, because employers have a strong incentive to hide salary band violations, while employees have a strong incentive to discover when negotiations are possible.

> it strikes me as an extraordinary to claim successfully negotiating higher-than-average wages is typical.

Then you should probably go talk to somebody who made that claim. But it's a nonsensical claim, so my guess is not many would be making it.

I think it has a lot less to do with being at the top of the skill curve than with making the business case for it.

I'm sure that's true. The world is not a technical problem. It just happens that the folks I know in that position more than deserve it from a technical perspective.

Yes what is and, according to the parent what should be, diverge.

What this really tells me is that workers in Omaha and Phuket ought to demand more, and that they are selling themselves for far less than the value they create, or else that they aren't actually doing "the same job." Likely both are true.

In most cases, it is at least partially tied to how much value someone can extract from you, which really has to do with how much value they bring in the market.

It's less about the 'value I create' and more about the 'value company X is able to create using my skills'. And... in some measure, you're going to be tied to how good the company itself is. I could give the same deliverable to two companies, and one may be able to make 20x more than the other, owing to other factors like network, sales, marketing, image, etc. In that case, they could probably pay me more as well, because they're earning more, but that then gets in to business vision and politics - do they see the software/tech as a cost-center or a profit-center?

Agree with this. Some in south India make 100K USD for the job that would make 140 to 180K in USA. But not everyone though and like other post mentioned it is about the companies perception of your value and most of the Fortune 500 companies have some base setup for each title irrespective of where you live.

I have direct experience with getting offers in an area with cost of living similar to Omaha. With no negotiation, the pay was the same as in the high cost of living area where I chose to live.

I don't have direct experience with Phuket, but I've heard of friends who get paid US level contract rates in similar areas.

I think it has a lot more to do with being a good negotiator and being good at leveraging your skills than being "really good" in a general. People don't get what they deserve. They get what they negotiate.

I believe that the wage pressure moves in the other direction: most Silicon Valley firms prefer local engineers, but they're a limited resource. If you're paying $2000+ in rent, you're incentivized to seek unusually high wages -- and, as a local engineer in Silicon Valley, you're in a position to do so. Employers are competing for /you/.

(Of course, we shouldn't dismiss the less market-determined psychological factors at play: if you're an executive at a startup flooded with easy money in a culture that doesn't particularly value corporate fiscal responsibility, you may want to pay better-than-market for the pure gratification of doing so.)

Most startups, including funded startups, seem to pay significantly lower than established tech companies in this area. My anecdotal experience certainly fits that. The last time I was job searching the highest offer I got from a startup was $130k plus a hiring bonus and laughable equity with poor benefits, while the young company (IPO within the previous four years) I accepted an offer from came in at well over 20% more than that all-in.

I think startups undercompensate and more or less target lottery-equity chasers.

I can't speak for everyone on the project—I've only really interacted with one person—but I just wanted to say that $200k all-in is an absolute bargain for him (if that's in fact how it breaks down) given his skills, experience, and ability to operate/drive the project.

Would their staffing cost also include office space (even remote companies have headquarters)? Or would that be included somewhere else? Or do they actually not have an office?

Complaining about that is insane.

To put it in perspective, almost any Fortune 500 is paying $200k for a contract Level-2 helpdesk analyst.

Short term contracts might be different, but 95% of normal software engineers in germany will e.g. never reach 100k. Including those in the large corporations. I think 200k might be barely in reach for a senior manager. In other countries/regions this will even be far lower.

It's a nonprofit looking for next years funding. An engineer defintatly ought to be paid as if it's a short term gig.

This might be part of the advantage America has over some European countries when it comes to elite talent.

Germany has horrible pay, a senior software engineer for Amazon and the likes is making 60K Euro's....

No one I've ever known makes that in support roles. Consultants can get that easily though.

The worker makes $30, the contractor and body shops take the rest.

What does it mean to analyze a helpdesk?

For more context, consider (1). It shows the expected pay for someone with a masters in comp sci, working as an engineer in Belgium. It tops out at just over 5k (gross) a month, so just about 65k a year. Less than one third of 200k.

(1): https://imgur.com/a/zRWXT

> To everyone complaining about the $250K (see edit, it's actually 200K) per person in salary

I don't think people are _complaining_ about it, just expressing surprise/shock. It's one thing to read the anonymous "I know a guy who's brother's roommate makes $300K at Google" comments that you have to take with a grain of salt, but another thing to see an actual company's data. Eye opening for sure.

$200K per person staffing budget isn't $200K per person salary. Benefits, employer share of payroll taxes, and other things that don't show up in top-line salary are in there. You are a lot bigger cost to your employer than your gross salary.

Yep. Here at UC, the non-salary costs can be in excess of 60% of salary.

And in the federal govt it's at least that percentage... Anecdotally, in one department within the DoD, it's ~90%, which sounds insane.

> Anecdotally, in one department within the DoD, it's ~90%, which sounds insane.

If you consider how little uniformed military personnel are paid and how high the other costs associated with them are to start with, I'm actually not all that surprised that somewhere in some specialized area in the DoD this ratio is reached.

This was interesting when I was in. I was making $25k a year as a corporal, but I was saving almost all of it because I wasn't paying for rent, utilities, health insurance, food, or the gym. My expenses were Internet, cell phone, and car insurance / gas. That was it.

I make three times as much in the civilian world, but I'm saving less.

They also probably don't include BAH and BAS as salary since they are considered reimbursements. Just as you wouldn't include cost of barracks as part of your salary if you lived there. BAH and BAS are a very large part of compensation.

We can take that to an extreme: <br> (from 2015) http://time.com/3651697/afghanistan-war-cost/

"...the CRS (Congressional Research Service) report says the cost of keeping a single American soldier there (Afghanistan) this year is an eye-watering $3.9 million..." <br> I don't think there's too many millionaires coming back from Afghanistan.

"Each Javelin round costs $80,000, and the idea that it’s fired by a guy who doesn’t make that in a year at a guy who doesn’t make that in a lifetime is somehow so outrageous it almost makes the war seem winnable.”

- Sebastian Junger

Just to add on, usually salary is ~3/4 of cost to the company, so ~$150k is the average salary which seems completely inline for quality talent with experience

Yeah you have a point. I guess living here and having many friends who make more than 300K/yr as engineers (and some as much at 450K or 500K, and a few getting >1.2M), I've become numb to it.

Since we're talking, how much did you make at Netflix? 250k seems "entry level" there for a good engineer. Do managers make more?

h1b data is a good source. Especially for Netflix since almost all their compensation is in salary, rather than benefits/stock/etc like a lot of other tech companies.

http://h1bdata.info/index.php?em=Netflix+Inc&job=&city=&year...

A good Engineer earning $140k as a base salary in SF shouldn't be a surprise (if it is, you're quite out of touch with the market) and then you have to add in the other costs companies have to budget for (benefits, stock, expenses, etc) plus it's clearly an average over the employees (so will include some that make much more and some that make less).

Honestly if it's surprise/shock it's _still_ a bit strange.

Why? I don't think it is surprising. Throwaway account, but when I interviewed at FB and Apple I got offers for over $1m/4yr total comp, which is $250k/yr. Is Let's Encrypt doing less valuable work? Should the employees voluntarily give up compensation for some reason?

My previous startup job was similar in salary to the FB offer, though without the guaranteed RSUs (>300k/4yr).

When I look at the public salary data it seems low

I think $200K/person is fine, but I do quibble with this:

> They are in Silicon Valley and have to compete with everyone else in Silicon Valley

Why? Why compete for talent in Silicon Valley when there are people all over the country who'd love to work on something like this? I can get why they might prefer not to do remote teams (it's a challenge), but there are cities like Seattle, Boston, Denver, Dallas, Kansas City, Houston — all of which have folks quite capable of building this.

In fact, for some folks not being located in SV is a huge plus!

As covered elsewhere, most of the team is remote. But they still get SV salaries because great people get those salaries no matter where they are.

It doesn't work that way for all the remote jobs and contracts I've worked.

Maybe you aren't a great engineer?

That's pretty insulting to suggest in any case and borders on a reprehensible attitude to take personally against someone.

I think what jlukanta is saying is that the greatest people get the salaries they demand regardless of location. If you're insulted by the suggestion that you might not be in that top category, maybe you're letting your ego respond instead of thinking about it rationally.

The idea that not being "great" is insulting is more toxic than assuming someone isn't great. Lots of hero worship in this industry that fucks with people's egos - your statement being a prime example.

He directly insulted me, clarify how in your thinking, this has anything to do with hero worship? Stating your views without falling back on the needless application of profanity is something I'd appreciate.

I'm pretty his/her intention was not to insult you. Since you won't even consider the possibility that you're not a great engineer, perhaps you're just bad at finding decent jobs and contracts? Or you're bad at negotiating pay? I'm honestly a little confused because my impression is that contractors and consultants make more money than full-time employees.

If you understand great as considerably better than average, then you probably aren't great. I'm not. Almost no one is, although almost everyone thinks they beat the curve.

If you understand great to mean merely competent, then I think you're abusing the meaning of "great".

I agree that it was a kinda shitty thing to say.

You offered yourself as the example. You don't have a lot of room for complaint when people look at the example you provided.

If it helps you unbunch your undies on this topic, consider that "great" in the original meaning was something like "world class". There's no shame in not being one of the top people in the world.

I did not mean that as an insult. I apologize if it came across that way. Others have made a better explanation of what I was trying to convey.

Or a great engineer that is not a US resident.

More likely just not a great negotiator.

Counterpoint: It did for me.

that is ridiculous, do you have evidence to back that statement up?

If you consider anecdotal evidence then I have received, with no negotiation, similar (statistically identical) offers in low cost of living areas and high cost of living areas.

I'd love to know where. Not trying to be snarky.

I work in a pretty small niche that's concentrated in high cost of living areas so I would prefer not to get too specific. One example was a company in the same country as Omaha with similar cost of living.

A different company was in a different country than Omaha and the naive cost of living was higher, but the cost of living, after adjusting for taxes, was actually very similar.

I hope that gives enough information. I really enjoy living in high cost of living areas so that's where I ended up, but I did have the choice to live somewhere cheaper.

Country, or county?

Country.

lol not ridiculous at all

Glad to hear it! That was my first thought on glancing over the figures: 'hmm, I bet they're located in Silicon Valley, they could save money by moving somewhere cheaper'. But if the money is actually going to hiring the best people instead of into the pockets of San Francisco landlords, fair play to them.

"Silicon Valley" is the red herring then. World class people get world class compensation because they are damn good.

I propose that not so good techies may get more in SV but paid averagely elsewhere?

Regular people get world class compensation (which this is not) because they ask for it.

I'm guessing part of the negative reaction you're getting is because great people don't just "get SV salaries" no matter what. To get an SV salary, you need to find employment at a company able and willing to pay an SV salary. Being a great engineer does help a lot, but you need to chase the salary, it won't just fall into your lap.

It's much harder if you want to be remote, and very hard if you want to live in a country where no company pays SV salaries.

Yea, those SV salaries are what a lot of CEOs make over here in Southern Europe. Of course, this might say more about Southern Europe, but for most of the world, when we see 200K a year paychecks, its hard to imagine that its for an engineer (especially for a programmer or sysadmin).

Also there are other countries than the US

There are not insignificant complexities involved in hiring people multinationally.

Is there a business op in this? Just a simple umbrella company in the US that supplies talent from around the world?

You people are hilarious.

Hey, YC, I've got a great idea for a startup in the incredibly hot cert/encryption space. The team is world class and in place. The product is shipping and customers are doubling every 6 months. $8 million at $40 million please.

Y'all idolize the bloodsuckers in Teslas that run this site, and give Josh and team grief. Jeez.

I somehow can't imagine 'dang owns a Tesla. Seems a like a Boosted Board guy to me. Also, not a bloodsucker.

I'm not giving Let's Encrypt grief — I think the wages they are paying sound decent. I was quibbling with the person I was replying to.

You are taking this way to harsh, I don't think he is giving them grief.

You would think people working for a SV company in places with a much lower cost of living would take a much lower salary, but you would be wrong. It is certainly my experience in hiring.

As it should. Nobody is going to get a discount on great engineers because everyone wants great engineers, and there are a hell of a lot fewer great engineers than job openings for them.

If the employees lived in Silicon Valley they would probably demand more than 200k/year for the same skill.

I'm a 3 years software engineer in France and I'm getting 29k€ (tax excluded) per year. With current exchange rate that makes SV guys getting paid 10 times more than me. Where's the onion ?

This just means France has a weak software industry.

> I'm getting 29k€ (tax excluded) per year.

Ask your employer first how much it costs to pay you that. It's probably much more than you think. The LE people get paid <=3x as much as you and that's easily justified with SV salaries in general and their expertise.

Lets be generous and say the employer pays 45k, its still way over 5x more.

Hi fellow French engineer. You are paid 29k€, which already covers for your health insurance, retirement plans, unemployment insurance, education, and so on. It is often said that in France, the employer pays 2x what he pays you (look up "salaire super-brut"). In America, you often have to cover your ass by yourself --and from what I understand, you absolutely don't want to get cancer or any serious illness there, no matter your health insurance. Also, living in these areas is fucking expensive -- a friend in NY made $5k a month during his internship, but had to pay $2k just for his apartment. The same holds in SV, I'd say.

However, I don't know your job but 29k€ seems a bit at the lower end of the scale to me, at least for IT. Have you looked at websites like glassdoor.fr what's the mean salary for your position? From what I know the first years are highly dependent on your diploma, but after a few years it shouldn't be effective anymore.

Edit: and as @chrischen says, IT engineers are not much valued in France. I'm pretty sure you've heard about the expression "pisseur de code" (literally, someone who pees code). I'm also pretty sure you've met "entrepreneurs" who had a "great idea" and who "only need" an intern to implement it --but surely they got the idea, the IT guy is just an operating man so he shouldn't get credit, even though is idea is 100% about IT services (and, probably, shitty).

> tax excluded

Why?

I thought it was totally reasonable considering the domain their work exists in (cryptography and security), the average salaries of professionals in our fields (especially in SF), and the overhead of benefits and taxes.

Really happy to see a member from EFF and another from Mozilla as well, two very respectable foundations in the fight for user and citizen privacy in the digital age.

I shall absolutely be donating in the future. (Especially since a 10 dollar per month donation is less than two ssl certificates at godaddy.

Love your work!

$200k, including benefits, employment taxes, etc. is probably more like $150k in salary. It's not at all egregious for good programmers. I've had employees make near that in Ohio and have friends in the field who make considerably more.

Also it's a budget, so I suspect it's intentionally padded to a small degree.

While prices haven't converged on SV everywhere, it does have a ripple effect.

You're overestimating their final salary. Per-employee expenses are usually almost the salary all over again. Payroll taxes are a real bitch. It's probably closer to $110-$120k salary per employee.

"Payroll tax" (more appropriately known as the employer-side of the FICA tax) is only 7%, it's really not what I would qualify as "a bitch". My wages are 80% of my total compensation (my employer provides a nice breakdown of insurance costs, 401(k) match, etc), factor in 7% of my wages going to FICA and their total cost to employ me with my total compensation package is ~125% of my annual salary.

I also make a fairly lower wage than the industry average since I'm not in SV, insurance rates don't increase proportionately with salary so if this is what I'm seeing at $75K/yr then at $200K/yr gross these employees are most certainly seeing at least $150K/yr in their salaries.

Not in CA. http://www.adp.com/-/media/Compliance/2016%20Fast%20Wage%20a...

Ah, yes, forgot about UI and workers comp, so let's make that ~130% (assuming an employer doesn't have egregious numbers of unemployment claims which cause their UI to skyrocket).

In my market (Ohio, US) loaded cost of an employee is between 130% and 135% of salary.

No. I've employed people in California. You don't pay 70% extra. It's more like the 25-30 range.

The human cognitive measure for egregious pay is not: Pay > Marginal Utility to Humanity, otherwise there's millions of jobs that provide Pay > MuH or even where Pay > 0 > MuH, and I don't hear people complaining about that fact very often. If you look at it through this assumption that compensation should be provided in exchange for providing a benefit to the world, complaining about LetsEncrypt dev salaries begins to look ridiculous.

That's a pretty damn high salary regardless. But I think that's fine if most of that money end up paying for the engineers who do the actual work and not the management.

You have a general problem with engineering management?

Just like the old "if you don't get paid enough, go somewhere else" mantra, if your manager is shit - go somewhere else.

If they provided an identical quality of service with 25 employees for $2m nobody would bat an eye. I'd say the fact that the team is so lean is justification for higher salaries in and of itself.

I think this is a better point. They seem to be incredibly capable for such a small team with such a large impact.

Could they provide a better service with 25 instead of 10 for $2m though?

Assuming the salary cost scales with the skill of the employee linearly and assuming that 25 people causes extra management overhead I would say no.

I think $2.91M is a modest (if not lean) budget for the important work they're doing.

How many websites now have HTTPS because of LetsEncrypt? How much would they have needed to pay for certificates from an incumbent CA?

I agree in terms of results and like them. Yet, I'll ask you since you're still in industry: "Wouldn't it be cheaper if they got hackers in Germany like close to Berlin? And not be operating in a police state with secret courts and such?" The latter is ironic given Germany's history.

Legal aside, I've seen plenty of folks over there putting together OS's, VPN's, cryptophones, etc. Most doing heavy lifting don't make SF wages or have office costs that high. Plus, one could poach some veterans in PKI from one of the big security companies. At least one of them. I doubt it's the only such place given all the country names in my files on cutting-edge research.

> Most doing heavy lifting don't make SF wages or have office costs that high.

I work remotely from home and don't make anywhere near as much as someone in the valley. However, I'm quite happy with my cost of living. :)

I see you playing it smart. Good for you! :)

I agree completely, especially regarding benefits. A lot of folks will look at a base pay rate and turn down the job because they will have $1000/month less "fun money" than if they take the job across the street. What they don't understand is the job across the street has shitty healthcare plans, little or no investments/stock options/pensions, less or less flexible paid leave, and so on.

I went back into government work specifically for the benefits, even though the pay was no more than I was already making at a private company. That company wasn't offering me any health insurance and I had to fight tooth and nail for any time off, paid or otherwise. At my current job, while it's not as mentally stimulating, I can retire at 50 years old with my sanity, full benefits, and retirement intact, and work for myself in my golden years without fear of losing health and life insurance.

Forget the salaries - you're getting REAMED by your laywers! $350,000 a year for legal and administrative is ridiculous, especially given that this is not for your salaried employees.

I agree and want to corroborate from another point of view, speaking as someone that's hired at and been a technical lead at multiple organizations: it's very hard to find actual talent, and paying a higher salary for one actually talented engineer is almost always a better investment than paying average salaries for, say, three average engineers (conservatively).

From that point of view, which I am convinced of, I not only see no problem with this - I see it as a good sign, and have more faith in the organization because of it.

Seems totally reasonable, also I don't know about the US but in other parts of the world there are other kinds of staff related costs you have to consider too like fringe benefit tax etc. Seems like let's encrypt is a great meaningful endeavour that produces good results for the world on a pretty elegant idea and lean implementation.

Considering what this team is doing, creating, and providing to everyone free of charge-- especially keeping in mind the scale of it, and the magnitude of it's impact on the entire Internet-- I'd say pay them more than $200k. They maybe should offer top tier salaries, in order to attract and keep top tier talent (not meaning to disparage the current team in any way, they could be best in show for all I know -- what they've done thus far is pretty incredible and they're just getting started)

According to https://letsencrypt.org/jobs/

"All of our positions are remote, and we can only hire in the United States and Canada."

Curious how benefits are worked out for remote employees?

Also its ~200k per person, which is all the costs including overhead: salary, benefits, equipment, etc.

There is an "Executive Director" mentioned in the list of employees. I wonder if that persons's salary is really the same as the engineers'? ;-)

Funny to see people quibble over a few hundred thousand dollars when they're doing a substantial amount of the heavy lifting to encrypt the web. At over 10 million certificates issued in the past year, their staffing costs represent $0.20/cert. For those who've deployed with Let's Encrypt, did you get a quarter's worth of value from their service?

To put in other words, if one considers donating a fraction of what would have been charged by any other certificate authority then we can be sure that LetsEncrypt would be able to continue to create a more secure web for all of us.

> For those who've deployed with Let's Encrypt, did you get a quarter's worth of value from their service?

The certificate price isn't even the issue. It's the signup + auto-renew!

Before LE came out, I would have paid $100 to not have to 1) Not go through the hassle of manually ordering certs and 2) Not having to log back into servers to periodically update them.

I get both of those with LE ... the fact that it's free? That's just icing on a very delicious cake!

How about donating $100 to Lets Encrypt then? :P

> How about donating $100 to Lets Encrypt then? :P

That'd require me getting a refund :P

After reading your suggestion, I just donated $200--it was $100 but I stepped in for koolba

Also if people don't like the Let's Encrypt budget, they don't have to donate, and they certainly don't need to use the service.

The important thing, to me, is that Let's Encrypt is being open and honest about how the donation money is spend.

Fascinating. Everyone here is complaining that 2M for 8 employees seems steep, but I think this seems really cheap. Admittedly, I work at Google, but we bat around the number of 400-500k per employee when discussing them here. 250k/ in SV seems really reasonably priced in comparison.

Maybe SV is not the best place then. Especially when they are working remotely. One thing is to make money and pay salary as you want, and another one is "we want such salaries, please donate".

But don't get me wrong, I'm going to subscribe for monthly donation to them. They save a lot of my time monthly.

I'd say it's a true statement. The reason you start a company in SV is to get access to the people who live and work in SV. If you don't use that pool of resources, there is no point in being in SV and paying to be in SV. Might as well move to Madison, WI and halve your expenses. And there you'd have access to University of Wisconsin graduates and can compete on wages with Epic instead of Google and Apple.

400K per employee at a 32.7 billion dollar company is a vastly different ratio compared to spending ~250K per employee at a start-up with limited funds.

Both are competing for the same talent. Also in the Let's Encypt case there are no stock options to make up the difference with the big tech companies.

Yes but different people have different tastes, and some may prefer a greater and steady base pay over a potential payout at the end that you have to have enough money to pay the tax on beforehand, or when cashed out. Anyway financial prospects aside, some may also prefer the flexibility of making an impact at a smaller company vs working for a bigger one. Same talent, different preferences.

Are you sure those figures are something your employer is okay with being shared here?

Google's publicly held. You can find out a lot more than that tidbit in their public filings with the SEC.

But not specifically a job title's internal expectation of fully-loaded cost or willingness-to-pay. You can find salary and benefit aggregates, sure, but that's not the same thing.

> I work at Google, but we bat around the number of 400-500k per employee

That sounds astronomical even by SV standards. Perhaps I'm completely out of touch.

When considering the salary for their staff, it is important to point out that LetsEncrypt only works if there is trust. They need to have high caliber people that they can trust and this costs money. Hearing that LE Is paying their people a fairly good wage, makes me much more comfortable using and supporting them.

Lots of people are talking about how expensive Silicon Valley is and how they have to compete with other salaries... but this is the justifying reason to me. If you pay enough that money is off the table, you can trust them a lot better.

They are not paying $250k per employee. You have to factor in payroll taxes, paying accountants to keep books and CPAs to do taxes, benefits, and a multitude of other regulation-imposed payroll costs.

All of this is to support salaries of around $125k - $150k.

The same goes for pretty much any employer. If a fast food restaurant hires someone at $8/hr, it actually costs them around $16/hr to hire the employee.

In a free market Let's Encrypt could probably cut its expenses to close to half what it's at now.

The post says that the Linux Foundation handles administration and accounting for them.

But I assume payroll tax is by far the biggest amount.

In a "free market", I'd demand higher wages to cover medical expenses, my membership in the local road association so I could use the roads, etc. And I'd get it because I'd be in a union (as a programmer in this current non-free market, I don't feel super compelled to unionize, yet). But otherwise, yes, agreed.

In a "free market", I'd agree to do it for the same minus the local road association costs. Sorry, you're out of a job.

Edit: People downvoting reality. OK.

I'm only out of a job if you can successfully get to work without using the roads owned by the local road association. And if you can do that, so can I, so I wouldn't be paying that cost. But this is SF we're talking about, so that seems unlikely - transportation is extremely monetizable.

Alternatively, maybe you're taking a worse route to work and willing to suffer a worse commute, in which case fine, you're just willing to work for lower pay than I am. I'm only out of a job if you're an equal-quality employee despite being willing to work for lower pay, in which case one of us is being stupid (either you are undercharging, or I am overcharging) and could just choose to stop being stupid.

In Sweden the head of state earns less money than the average Let's Encrypt employee.

The Swedish head of state does not live in Silicon Valley (and I would be surprised if the Swedish state is spending less than $250k/yr. in total on its monarchy).

My point was not that Sweden i great and cheap, it was that you can get allot for those sums of money.

Good, since LE employees are actually providing value to the world.

The President of the United States earns less than some federal employees.

Same here in New Zealand, the Prime Minister earns less than several public servants.

I'm quite surprised - $2.06 on staffing seemed steep, so obviously read on as I was thinking that there must be a lot of employees. But 8? OK, I'm just a teacher, and I earn around £30k a year... not the kind of figures I've seen bandied about on here for graduate programmers who seem to be pulling in 8-10 times that for their first job... So I'd be interested to see a breakdown of the payroll (which I doubt would happen) - an average of $250k seem normal to others on here?

people who seriously know how to code mission critical X.590 root CA PKI infrastructure are not cheap when you're looking at total benefits/compensation per head.

>people who seriously know how to code mission critical X.590 root CA PKI infrastructure are not cheap

And they also aren't working at letsencrypt.

You've exhaustively enumerated those people, have you?

I know a couple folks at Let's Encrypt in passing and they are at the top of my list of folks I would want writing code for me if I needed to trust it with, if not my life, most stuff up to that point.

I'm not saying they're right, but it's not hard to enumerate 3 or 8 people.

Ok, but how little you value your life and how little you know about secure coding isn't useful to the rest of us.

Creating a certificate is easy.

It sure is! Just go to https://letsencrypt.org

It's so easy, I have a cron job do it for me!

(That is a comment on how easy LetsEncrypt makes generating certificates, something I very much appreciate).

I agree, LetsEncrypt has done a great job. My original comment was snarky though accurate in a focused sense concerning the actual creation of a self-signed certificate (which can be easily cron jobbed). Of course I understand and appreciate that there is the whole negotiating the larger certificate scene, multiple companies, politics, etc. and that takes time, talent, effort and money. If LetsEncrypt had just come out from the start and said, "we can offer this service for X dollars" then it would have been more transparent.

Creating a certificate might be easy for someone with the proper skillset.

However creating an world-class organization that provides certificates as easily as LE does... definitely not.

How about creating hundreds of millions of certificates, developing a new standard protocol to interoperate with CAs, managing millions of registered users, managing certificate revocation, logging to certificate transparency logs, running OSCP responders, developing clients for a wide variety of platforms, and keeping up with the latest happenings in the crypto world?

Have you tried writing that code from scratch without any libraries?

I guess 20k a month per engineer by default, all in. That's probably out of date since I've been using the same number for over ten years now.

With things such as tax and healthcare it typically costs a business 1/4 to 1/3 more per employee than their wage, so each person here is perhaps on $180-200k assuming a completely even split. Considering this is deep crypto in San Francisco it's not too surprising.

> deep crypto

I'm sorry but this isn't "deep crypto". Very technical jobs that need a very good job done, since a lot of people depend on it? Sure. But they're not inventing anything new here. With all due respect, signing certificates is something you can grab from the openssl library, rewrite in a memory-safe language, have the community look at, and you're good to go.

I'm not saying I think 200k is too much. There are many reasons why it adds up to that much. I just think the technical requirements are not the reason for those wages.

Yes, that seems about right. I also do not get paid nearly that much, but the location (San Francisco), caliber of required employees and the overhead of benefits certainly adds up.

> an average of $250k seem normal to others on here?

For the work that they are doing (including the overhead of an employee -- health, benefits, taxes, etc), this seems cheap.

"plus two full time staff that are employed by other entities (Mozilla and EFF)"

So, that's 10. $206k/employee The accepted amount of benefits & costs per employee is around 35%, so now we're at $134k/employee. This is not a high salary for an engineer in San Francisco. In fact, I'd say it's pretty damn low.

LE isn't paying for the staff coming from Mozilla and EFF. The 2.06m is split across 8 people, not 10.

EDIT: Per a follow up tweet, the staffing is expected to increase to 10 in 2017 (presumably plus the two from third parties), so it is /10, not /8, but not for the reason you described.

True, I misunderstood who was paying the salary. Even at 8 it's just a normal SF salary.

A fully loaded cost of ~$250k for a senior developer or SRE is not at all extraordinary. Especially when for something like Lets Encrypt you really want/need to hire experts.

A note: You don't cost £30k to your employer, more likely 1.5-3x that. So say an employee costs a total of $250k, their pay could be anywhere between $80-180k with other compensation (retirement packages, health benefits) and overhead making up the rest. (the higher the multiple, the less efficient the company likely is, in this case I'd expect it would be closer to the 1.5x mark than the 3x mark)

Going to repeat this comment here since people are doing the math: That's for ten employees, not eight. Salary + benefits.

Could have been more clear in the post initially, but we have eight employees now and have budgeted for ten in 2017. I added a sentence to the blog post to make this more clear.

I suspect salaries for software developers in San Francisco are just a bit mad. The numbers floating around suggest more than twice what you'd get paid working on hedge fund software in London, which is in turn maybe 3 times more than an ordinary software developer elsewhere in the UK.

(If it makes you feel any better, I earn less than you working as a software developer in the UK.)

The hedge fund and biz companies are not getting the VC money chasing the unicorns, whose crazy valuations are propped up by cheap money.

No, it is absurdly high. They should be paying half that.

Programmers and teachers are not really paid well in the UK !

Your 30k salary costs your employer around 50-60k (I don't know the deal with your benefits and payroll taxes and all that).

Rule of thumb in the US is that you add about 40% to a salary to get the total cost to the company to support that salary. And that's just taxes and basic benefits. Add a pension and 401k with matching, and it goes higher.

The pay discrepancy between teachers and programmers in San Francisco is really bad.

- School districts in SF and Oakland are short dozens of teachers -- imagine having sixty classrooms on the first day of school where there is no teacher. (e.g. http://www.sfgate.com/education/article/Bay-Area-school-dist... from last year)

- San Francisco is basically building "dorms" for teachers to give them a place to live: http://www.sfgate.com/bayarea/article/Mayor-and-SFUSD-have-a...

- Many teachers are leaving the area to go work in more affordable parts of the country: http://projects.sfchronicle.com/2016/teacher-pay/

> The pay discrepancy between teachers and programmers in San Francisco is really bad.

I'd say, 'reflects the distribution of skills and positions available' rather than 'is really bad.' There is at least an order of magnitude more people capable of being minimally-acceptable public-school teachers than are capable of being minimally-acceptable Silicon Valley programmers.

That's a fascinating assumption.

Why is it a given that teachers "must" have low salaries? The sixty classrooms without teachers show otherwise.

SF teachers clearly need raises, and those can be funded by taxes from all those programmers that presumably benefit from having schools in their communities.

In fact there is no reason why teacher wages shouldn't keep up with the general state of the community where they live in.

> Why is it a given that teachers "must" have low salaries?

I explained why: at least an order of magnitude more people are capable of meeting the minimal requirements to be teachers than are capable of meeting the minimal requirements to be developers, thus the former will always be less-well-paid than the latter.

> The sixty classrooms without teachers show otherwise.

They may not be due to pay but due to union rules. And, of course, even if teachers should make more that doesn't mean that they should make much

> In fact there is no reason why teacher wages shouldn't keep up with the general state of the community where they live in.

Are you saying that teachers should make a mean or median wage? I think that's a little silly, as it's hardly a profession requiring high innate talent and intensive study, unlike engineering, medicine, law, religion. Frankly, I think a welder or a plumber should be paid more than a teacher.

But it shouldn't matter what you & I think teachers should be paid: their wages should be set by the market, as should everyone's. That's the closest thing to fair we'll ever be able to get.

All the complaints about salaries are dismaying. Why not consider that plenty of other places seriously undercompensate engineers instead of trying to cut this organization's pay structure down?

People overwhelmingly want others to have less than them more than they want themselves to have more. It's ridiculous, but true.

Yep. The same phenomenon augmented by racism applies to certain demographic populations of the US with respect to minimum wage, welfare, etc.

It's just completely irrational.

> We’re currently working to raise the money we need to operate through the next year. Please consider donating or becoming a sponsor if you’re able to do so! In the event that we end up being able to raise more money than we need to just keep Let’s Encrypt running we can look into adding other services to improve access to a more secure and privacy-respecting Web.

Wouldn't they want to secure funding for 5+ years for existing services before considering adding other services?

I'd guess that the company-sponsors are on the hook for more than a year.

It is astounding to see so many people here who think employee cost == employee salary.

To Let's Encrypt: if you guys took bitcoins as donation I would give $10 in an instant. The friction of filling out this long Paypal form is just too high for $10... http://i.imgur.com/oZZWeuG.png (And, no, I don't have a Paypal account.)

Make it easy to give, and you will receive more donations!

Edit: I am not the only one who wants to donate BTC! Bitcoin was the number one request asked last year: https://twitter.com/letsencrypt/status/581177744152784896

I'm in the same boat, though in my case I'd prefer Stripe for card processing rather than Paypal.

Stripe already has my card on file from other sellers, and I trust them where I don't especially trust Paypal.

Why would you have any less reason to trust PayPal to process a credit card than Stripe or anyone else?

That's an interesting way of putting it. I think I would spend longer double-checking a bitcoin transaction that I would one through paypal.

Name/address are easy to autofill even if you don't want anything saving your card info.

I'd do the bitcoin donation as such: unlock my phone, launch Bitcoin Wallet, capture on-screen QR code, type $10, hit send. Done.

I guess that's simple enough if you have two devices handy at the same time. But the simple flow for paypal is just as easy. Click to let my browser autofill payment info, type the CSC, hit submit.

Unfortunately, the historical precedent is that demand for bitcoin payments in polls and online forums is much greater than the actual demand if they actually go through the trouble of accepting accepting payments.

> The friction of filling out this long Paypal form is just too high for $10

You have got to be kidding me.

$20m is such a great bargain to how much value LetsEncrypt provides to the world at large. I can hardly think of better money spent.

They have a donation page here: https://letsencrypt.org/donate/

Let's keep it going.

I wish there was a way to donate money towards a fund - like, give $100USD, that goes into some index fund, and then LetsEncrypt withdraws $3.50 (as per https://www.bogleheads.org/wiki/Safe_withdrawal_rates) a year until the sun burns out.

I'd much rather spend my money knowing its a small ways toward making this kind of thing self sustainable.

You could set such a thing up yourself. It's basically a trust. Put your money in the index fund, and have a process to automatically withdraw the safe amount and donate it.

Now build a product around that idea, and be the next patreon. I heard of this great company that'll help you get ssh certs for free.

It'd be a stellar price list:

- $100: Donate $16,100,000,000 ($3.50/year for 4.6 billion years)

- $200: Donate $32,200,000,000 ($7.00/year for 4.6 billion years)

I've thought about this extensively, and it's actually not that easy to set one up yourself. For one thing, the overhead required to administer a trust is non-zero, and would be prohibitively expensive (non-sustainable) for a small amount. The barest minimum endowment is at least $10k, and possibly $100k. Best case, you find a way to align the trust with a university, who adds it to their portfolio and administers the funds according to your/their goals.

This also presumes that there are public investment instruments which can get a reliably positive rate of return. In today's economy, it's difficult to get even a few percent reliably. Some places are toying with negative interest rates, even (which means that passive investments are deflationary).

How does the overhead scale when you go from, say, having $10k in the pool to $10m? If a service existed that ran a trust where you could put in money and then dictate where your share gets routed, would this overhead keep going up dramatically as it grows?

The overhead is nearly constant with respect to the number of dollars. But the cost goes up linearly with the number of shareholders (as each one needs at least a modicum of attention over time, to update records reflecting changes of dispensation, or to verify death certificates, etc). And the costs also go up with the complexity of the trust, e.g. if the money has to do anything other than being held and dispersed at appropriate times. (Someone has to file the taxes).

Banks often employ someone called a Trust Officer, who handles things like this related to the trusts that the bank is managing (and carrying). They make around $30/hour. And let's say that 100 people each put in $1000, or $100k total. At 3% interest, the fund would generate $3k/year. If each shareholder requires an hour of the trust officer's time over the course of the year, then all income from the trust goes to pay the trust officer, and there's nothing left over to be dispersed.

It sounds like a good idea, and I'd definitely take advantage of the service if it existed. But lawyers and accountants have to be involved, and they want to be paid. And they're expensive.

If Let's Encrypt believes that's the best way to spend their money, they'll do that. Do you think you've thought more about their sustainability than they have?

I'm not talking about how Let's Encrypt spends it's money, I'm talking about how I spend mine :)

Assuming Let's Encrypt knows more about its sustainability than you do, and given that they can also invest their money in index funds if they believe it's the right approach, isn't giving the money to them outright just better?

The services of Let's Encrypt, let alone the cause they stand for, is well worth a small recurring donation. I think this is totally reasonable to pledge even $2/mo.

If companies that uses Let's Encrypt would chip in just 1/10th of what they'd pay for certificate from for profit providers, then Let's Encrypt won't be in need of donation anytime soon.

ICANN could just fund the whole thing.

That'd be a bit ironic, ICANN funding a project to verify assigned names and numbers because decades ago folks decided they couldn't trust ICANN to do its job.

I recently replaced my paid SSL certs using LE. I took what I would have paid in SSL costs over the next two years and donated it instead.

That's a fantastic initiative! Bravo!

So, what's the plan for when the bubble pops, Let's Encrypt runs out of funding, and meanwhile all browsers have switched to requiring HTTPS for all websites?

What?

I'd assume people would have to start paying for certs.

Resume, not start. But yeah, simple as that I'd say.

But I very much doubt it will get that far: the only cost they can't cut back on is hosting. The rest remains as-is if you don't touch it. Those hosting costs I feel like we could easily cover with the community, but they even have big corporations behind them to drive it when need be.

My guess is someone like Google would come in and fill the spot for free or very very cheap. I think Google/Microsoft/Apple/etc could donate the 10+ people they need without even batting an eye.

And Let's Encrypt have created open-source software and standard protocols for running an automated, free CA. So anyone (with sufficient money) could provide a drop-in replacement if needed or desired.

So programmers mad that other programmers are making good money (although, taking into account overhead, salaries are rather normal). This is why we can't have nice things people.

"Hardware/Software" and "Hosting/Auditing" seem to be weird things to lump together.

I wonder what software they pay for.

Hardware and software are both capex.

Hosting and auditing are both opex.

Makes sense to me.

Doesn't to me. Staffing/Auditing/Legal/Administrative are up to 2.7MM of their 3.0MM expenses. The way they grouped things is very deceptive.

Funny to see that some of the sponsors on the Let's Encrypt website are linked without https even though they offer https.

My multi-domain (3) cert just came up for renewal at $28.99 a year, instead I setup Let's Encrypt on 8 domains for free. I think I'll just setup a recurring donation to Let's Encrypt simply because I'll never have to do the SSL Cert song and dance ever again.

Thanks for doing such a great job. It's truly amazing what you have achieved with such a small team.

GEEZ - Those Legal and Administrative costs are INSANE. Especially if a lot of their stuff is handled by Linux Foundation. What the heck? 12%+ of total spend to lawyers and admin, and this is not salary? What's going on here?

People are confused. The end salaries are probably in the $110k-$150k range.

Surprised no one is commenting that their software has really poor usability. It may work well in providing secure certificates, but setup is a PITA and the documentation is abysmal.

Agreed. If I remember correctly it even destroyed a site once when it was attempting to make sure their validator could reach the verification file. I just did what everyone said: "it's so easy, just run it on auto mode with apache and it'll configure your cert for you". Turned out to be a huge PITA.

Recently I had to renew that site again (I've been postponing and dreading it) and looked for a different client. The getssl bash script[1] had its own problems (all caused by me not taking the time to read the readme, but the default settings were a bit odd), but overall a 10× better experience than the official client. Much less magic, much more a unix-like tool ("it does only one thing, and one thing very well").

[1] https://github.com/srvrco/getssl

I'm not sure I like their "we'll do it all for you" approach. It seems like there are a lot of ways to accidentally break config files. In general, I'd rather not anyone touch those without me knowing what it will do.

Fortunately, the certonly script grabs the cert and puts it in an easy location. Updating the config from there is easy enough.

Really?? I found it dead simple to use.

1. Install the app. 2. Run the certonly script, following the prompts. 3. Point the server at the certificates.

The only hitch was stopping Nginx long enough to do the domain verification. A three line bash script fixed that.

Check out Lego [0]. You can do DNS verification instead, no need to restart nginx :)

[0]: https://github.com/xenolf/lego

Offering an alternative viewpoint. Many seem to be focused on the cost/FTE. The alternative view is that if this was a business then $3m a year to run it is "peanuts" for the services provided. In fact I prefer the for profit model. I'd like them to charge for each account/cert to cover their costs.

The whole point of Lets Encrypt is not to charge people for the certificates. But I agree that the amount they are spending per year is very modest compared to what they are doing.

So many people complaining 'why not remote', when half the staff is operations, and they run their own hardware.

It's going to suck when your ops staffer in Tahiti has to fix a physical server problem in the SF datacenter...

Ya'll should get some swag for donors. I'd like a cool sticker :)

Reading these comments, it's hard not to agree with the sentiment that being transparent about salary data is of dubious value to a company.

2m - ~200k in benefits. Looks like 180k per engineer. Thats high, but looks like they've got the corporate sponsors to support those salaries.

They need to spend less or they're the next Tor foundation, an ineffective black hole of hacker wannabes and hanger-ons. Running high performance services don't cost very much as evidenced by their hosting entry. Maybe 5 engineers max. No way they are working on features every day. Should be very slow outside of patches. CA/SSL specification does not change weekly...

> CA/SSL specification does not change weekly...

The draft ACME specification does though.

Has anyone tried to incorporate signed executable loader into the Linux kernel ?

Hopefully letsencrypt can start certifying code signing keybase keys.

Also would be nice if someone would offer a service to build executables from open source code and code sign certifying that a particular executable was built from a particular source openly available.

There is no problem with being located in sf but I would take issue with the idea its a job that 'requires' one to be located in sf or have access to the absoute 'highest quality' of engineers.

On the contrary there is a risk of adding complexity where none is needed because super high quality talent requires super high quality work and if the work of generating certs is not challenging enough, complexity is created, like Acme.

I am not sure why generating a cert is so involved with letsencrypt. Does Dns validation, file validation or email validation require a new standard. Perhaps a simple rest api and web frontend would be better.

Here is my domainname, my cert request and a file on my server or dns entry to verify, now can I have a cert please? This is the process. Why does it need a client, and one that can automatically update certs!! and an elaborate new standard? There is no way I am running this on a live server, so the automation is then moot.

I would rather pay $20-40 a year than babysit my ssl every 90 days, and this is for a wildcard.

The kind of folks who automate things spend hundreds of dollars a month on aws or gce, and willingly pay 10 times more for bandwidth than they would at do or vultr. Is $20 a year or $2 a month per wildcard domain in this context really an issue? So who does the automation target and benefit?

For others a manual process would have been more accesible, there are tons of others things to do with one's time, le is automated but that is one more thing to monitor and that can go wrong.

I can't help feeling the whole cert generation process could be simplified and artificial restrictions around 90 days and wildcards re-evaluated. That would be valuable to users, contribute to make running things simpler and money well spent.

If the market crashes or they cannot raise needed funds, they could easily turn around and begin charging $1 per certificate per year and still be a much better value than other CAs.

I would be interested to know how many certificates they currently issue. I would have no problem paying a small annual rate for using the service.

I notice that they explain the purpose and benefit to the organization of every single category of employee - except the executive director.

If any LE staff are in here, I'd happily chip in on a Patreon.

You can donate directly, either once off or recurring monthly - https://letsencrypt.org/donate/

Maybe I'm odd, but I don't trust Paypal with a recurring monthly debit.

Their track record makes me doubt it would just be smooth and painless.

I'd rather avoid PayPal, and I like Patreon as a central clearinghouse for monthly donations.

On the flipside, Patreon takes a 5% cut on top of whatever fees there are for the payment method you use, so there's less of your dollar reaching the intended recipient.

https://cms.paypal.com/us/cgi-bin/?cmd=_render-content&conte... says PayPal takes $0.30 per transaction plus 2.2%, so that's a full 1/3 of my dollar gone there.

At first glance, it looks huge. But then, honestly, when you just consider vast benefit all of us are getting from them, this is nothing and probably should be funded on a higher level. Like 10-15M.

Curious where the 20K a month in hosting is spent? Perhaps Google or Amazon could throw in some credits to help out?

It is almost certainly necessary for them to have dedicated hardware in neutral datacenters. They can't just run their infrastructure on AWS or Google.

Maybe there's something very obvious I'm missing - but why?

If their environments are not bare-metal then moving from a virtual solution to a cloud one seems fairly simple. If they are bare-metal then it hardly seems necessary for issuing certs - but if you have more insight I'd love if you'd share it!

CAs require special hardware security modules (HSM) to contain the signing keys and have to control the entire path to them, so yes, you need to own your hardware for a public CA.

And independence from their sponsors is an important organisational feature for Let's Encrypt.

The second part of it is why they can't use cloud HSM products offered by various companies. If you're functioning as a CA, you absolutely want restricted physical access to your root keys, which you can't guarantee otherwise.

Security sensitive workloads are often not virtualised because it opens them up to "What are the other virtual machines doing?".

There've been a number of Xen and KVM issues over the years, including a couple which may have resulted in a bad actor having access to the hypervisor (and thus, all its guests). Attacks like Rowhammer also play in this space.

How many employees does it take to run Let's Encrypt? 8 2 to run the website 1 to ask for money 5 to say "are you sure you need a wildcard cert?"

|$250k/employee

hmmmm

They should consider running the service in Canada, where software engineer / programmer salaries are a third of that

or 90% of the US that is not SF/Silicon Valley

But they'd still be competing with Silicon Valley for top-end developers, who are going to be asking for a salary equivalent to their skills. You might save a little once you factor cost of living adjustments, though you'll also have to offset that with increased difficulty in finding the right person.

Folks who know security well enough to do this correctly are rare. You don't want to farm out critical X.509 infrastructure to the lowest bidder.

What do Verisign, RSA, Comodo and GlobalSign have in common? Not being in SF.

I didn't say you had to be in SF. Just that you'll be competing with SF for talent.

A follow up tweet clarified that it was 10 people and not eight. But even that is a pretty high cost per employee I agree.

But the question is: Can you get a team of engineers that have the ability and crypto know-how to build something like Let's Encrypt in Canada for 1/3 of the salary?

probably not. the best canadian coders are already here in the US.

Says who?

Says a Canadian who lives in the US who overgeneralizes.

It's not entirely true, but its way closer to the truth than you'd admit.

I mean I am a Canadian who lives in Canada and I disagree. So that's why I don't get your argument? I know plenty of very skilled people here.

But you gotta admit a lot of recent grads end up leaving. I'm another Canadian who left given I was offered doubled my salary in one move.

But isn't the cost of living also higher and you have medical expenses and things? Are you actually taking home double as much, or does a lot of that extra have to go to living expenses that you wouldn't have in Canada?

I do take home a lot more than I would in the Canada. Cost of living in the US isn't 2x Canada but salaries for tech jobs are (And keep in mind that was a while back, it's now more than that). Health is covered by my employer, I pay nothing directly for it. (Definitively not the majority of cases, most people have some amount of contributions to it). But if I had stayed in Quebec I'd have been in the top bracket and paying a huge chunk in taxes. Even as I live in a state with high social programs (MA).

So to conclude, if you are in a field like tech, medicine, finance you will be significantly better off in the US than Canada. Other fields with similar income levels between Canada and the US I'd say Canada will offer a better standard of living.

The cost of living is higher. Medical expenses aren't really a thing for recent grads, and good medical insurance limits your outlay. Normally most people with solid insurance and serious health concerns will max out at a few thousand dollars a year. And for the kinds of jobs we are talking about, that isn't a big deal.

Anyways, I'm making way beyond 2x as much as I ever could have in Canada. Also I get to work on world class tech, with world class people. And when I changes jobs, I don't have to take major compromises. There are way more high quality employers in the bay area than anywhere else for technology!

I think it was kind of a joke with some seriousness to it. It's really easy for Canadians to move to the US to work. So a large chunk of the highly qualified Canadians do leaving less in Canada. Canada suffers a shortage of highly qualified people in many professions. For example medical professions have it really hard given doctors moving to the US can make multiples more than in Canada.

And that's why all almost all the software engineer / programmers leave.

You're forgetting about their health insurance, retirement, unemployment taxes, etc. I'm sure they get some tax-relief being a non-profit, but still, high end computer scientists are expensive.

Open source projects aren't really volunteer/spare-time/garage efforts anymore. The Mozilla and Apache foundations are pretty big organizations. A lot of projects are also run by for-profit commercial players like Redhat and Canonical.

Bitcoin?

no windows sadly

can you explain why I'm downvoted?

There are a number of client options for windows[1].

[1]: https://letsencrypt.org/docs/client-options/#windows

seriously has anyone actually got to work any of these third-party options? Don't they all depend on IIS which is not free?

Many of the clients above support something like certbot's webroot plugin, which is web server-agnostic and works by simply putting a verification file in your webroot and serving that to Let's Encrypt's validation server.

Also: As far as I know, IIS is included with any Windows version (definitely any version of Windows Server).

Can the business model be charging for the certificate directly, just cheaper than others? I'm fine with that.

I'm not. I test a lot of ideas but don't have much money. I have already to pay for hosting and DNS. SSL is optional. I'm very thankful to let's encrypt for the opportunity to use SSL on my sites. Also SSL (and the web) is a broken technology, I'm not going to pay for something broken.

there is nothing really that is free beer in the world, the money has to come somewhere, either some "rich" guys donate the fund, so the rest gets a free ride, or everybody pays a little, say, $5 per year, what's wrong with that? I'm not a rich guy per se, but I want to pay reasonable fee for what I got.

You can still do so by donating 5$ per year or any other amount you'd like - which is awesome! More things should come around in this way. Talk about being inclusive.

yes but if you do the math since not everybody is paying whoever donates need pay more to take care of those who does not, which is really what the whole topic is about.

$2.06M for eight employees seems very steep

Approximately 100% salary overhead is pretty standard, so eight $125k employees plus infrastructure and office costs sounds reasonable.

8 x $125K would be $1M, not the $2M that is currently being spent on employees.

Edit: Missed the first part of that, I apologise. 100% overhead doesn't seem reasonable per employee to me though.

it's (8125k) 2 since there is 100% overhead

He said 100% overhead on top of salary

You missed the "100% salary overhead" part of the sentence. Consider taxes, benefits, insurance, other non-salary costs to employing people.

You missed the 100% overhead. Even 50% overhead at a slightly higher salary wouldn't be surprising.

Can you explain why there would be 100% salary overhead? This seems quite high as a non business owner/operator.

Taxes, health care, other insurance (disability, etc.), and other benefits (401k, etc.) are a significant fraction of the cost to employ someone. These things are not cheap.

The standard overhead is (I believe--correct me if I'm wrong!) anywhere from 25% to 100%, depending on your base salary and how competitive the benefits package is.

At our company the overhead is about 40% of the salary, and could easily reach 80-100% with bigger and better benefits packages.

Employer taxes, standard benefits (health, vision, dental, etc), extra benefits (like free food, budget for books, and other niceties), and office space.

Not really.

Overhead doesn't scale linearly with pay. Once you're into 6 figures, it's definitely not equal. I'm guessing much more like $150-200k per employee in compensation.

Is it really 100% in SF? That strikes me as high -- even in a lot of European countries it's not a full 100%.

Anyway: considering what they do and where they do it, I don't find these numbers high at all. I would be shocked if they didn't have at least a couple engineers making over $200K in actual cash compensation if they're working in the Bay Area.

My thoughts as well. Roughly 250k per employee?

If you want something this critical done correctly, is it really that steep?

I'd like to see it broken down further by staff, it would be interesting to see the responsibilities of the "Executive Director" and weigh up the salary difference between that and the median salary of the rest of the team.

250k per employee all-in (benefits, etc) doesn't seem that outrageous being in SF. Plus I'm guessing they're all high caliber employees given the scale of what they set out to do and what they've accomplished with a relatively small team.

"Being in SF" is a silly excuse. There is no reason for them to be in SF.

Except the people already were there, and they want to stay there. The imaginary high caliber people who can do this in a low cost of living place are exactly that: imaginary.

I fully endorse Let's Encrypt right to hire people solely from SF, but if you think high caliber people don't live in low cost of living places, you're either uneducated or purposely disingenuous.

EDIT: Source: I work on a fully remote team (>50 people) with teammates around the world who are high caliber people.

I think you can be more charitable in reading the GP's point; I certainly didn't interpret it the way you did or see it as a statement that high caliber people only live in SF or other high cost of living spaces.

It was a response to the claim that "there's no reason for them to be in SF" which is an odd assertion.

The point is that comparing an actual team with a purely hypothetical one is really just an exercise in imagination. These people exist, are competent, happen to live in San Francisco, and this organization has built a compelling product paying them what is likely lower than what they would earn at other employers in their region.

As I understand it nothing is preventing another service from emerging that competes with Let's Encrypt and has a better cost structure. It may also be reasonable to refrain from donating to / funding Let's Encrypt if you feel that they are poorly managing their finances, but even that to is more productive than simply stating "this team should not be in SF".

Exactly.

The reality is everyone is playing armchair quarterback/back seat driver/whatever and telling LE about that they COULD do. As if somehow LE was dedicated to overspending. As if they didn't scour the earth to find the people and save 25-50%.

Building a good team is hard. Building a good distributed team is harder. That they exist is just as much luck as anything else.

Pull open your brower's cert store and have a good look around. I suspect you'll discover that the majority of people who can run a certificate authority don't live in Silicon Valley.

It's hard work. It's challenging work. It's been screwed up a few times, yes. But it's not the sort of thing that's so hard that only five people in the world can do it correctly or something, though.

They have to compete with the rest of Silicon Valley. That's about what it costs to get a good engineer around here, when you count benefits.

Why would they have to be in Silicon Valley?

They don't (and in fact I think they are mostly remote) but they still have to compete with it.

Correct. LF has a small office in the Presidio, with perhaps a half dozen people living in SF proper and another dozen in the greater Bay Area. The rest of us are scattered around the US, Canada, Japan, and Australia.

just because LE competes with Silicon Valley in recruiting doesn't mean the employees are, or need to be, in Silicon Valley.

Couldn't this be ran out of Nowhere, Idaho though? Remote workers, etc.

I assumed they were remote, otherwise I would have thought they would have premises costs stated. Rent, heating etc.

That's for ten employees, not eight. Salary + benefits.

Could have been more clear in the post, but we have eight employees now and have budgeted for ten in 2017.

Assuming 100% overhead for benefits, etc., that's 100K per employee, which is quite low for Silicon Valley.

The 100% overhead only applies to the "average" job. Health benefit costs are typically the biggest piece of the benefits puzzle and don't scale with salary. In SV the overhead is more like 25%.

Its actually 2.06m for 10 employees: https://twitter.com/0xjosh/status/778283999609298944

I know in addition to salary there are other "hidden" costs like health care etc, but even taking that into account their employees must be taking home a pretty significant wage.

Would the business perform any better if they were paying $2.05M for 20 employees? That's $100k per employee including overhead. Salaries would be significantly less than $100k. On paper it may sound better, but you end up with extra administrative overhead, and lower quality work. Could twice as many junior staff perform as well as that many senior staff? My guess would be no.

Their staffing cost probably is on the higher end, but I'm guessing thats by design.

Many people here don't realize what it costs a business, in taxes, to employ someone. I think if more people realized this, we would have less people demanding a $15 pay-raise for minimum wage workers.

I strongly disagree. What the worker needs to make to have a puncher's chance at a life is not related to the amount paid in taxes.

Yes it does. As a business owner, more taxes means less money for employees (and less employees hired).

People that don't understand taxes never consider this and immediately think employers are just greedy.

Inflation is also never considered, which is exactly why we have to keep raising minimum wage in the first place.

Minimum wage increase will not help anyone in the long-run. It will only inflate the cost of goods and services for everyone within a few years.

The real answer is more education

Let's be really real for a sec: minimum wage does not, in any meaningful way, drive demand-pull inflation. Minimum wage doesn't even track the rate of inflation. This is macro-averse horsepuckey.

This position of abolition of minimum wage is not seriously held by any macroeconomic experts except on the extreme far right. There is a reason that wage floors, in lieu of GMI, are accepted by even centrists: because the alternatives are literally unconscionable.

Further: capital is greedy. It's functionally unable to be anything else. Throwing labor a bone once in a while is not, desperate hair-shirting aside, going to somehow kill it and destroy the jobs, homgz.

This does not instill confidence in Let's Encrypt's longevity. It sounds like a cry for help. $250k may be a reasonable sum for an established company to pay it's employees but for a company to have that kind of burn rate with out having next years funding secured is scary.

Have you seen their sponsors page?

Multiply the cost of the various sponsorships by the relevant sponsors and I think you'll find they're doing alright.

That doesn't even take in to account individual donations either.

The point of this post was not just fundraising, it was because they believe in transparency and openness.

It reads like, oh shit, if you don't donate we might not be around next year.

Pro Link

<a href="http://www.indo-news.ga">Indonesia News</a> <a href="http://www.tutorialgayarambut.com">Hairstyle</a>

Reply on news.ycombinator.com