How I operated as a staff engineer at Heroku | Hacker News Reader

How I operated as a staff engineer at Heroku

amyunger.com

380 pointscraigkerstiens84 comments

A thought I had recently is that staff/principal engineers who got their position by long tenure (not hired recently) should delete more code. Deleting code is actually one of the most difficult things to do in a project. To do it properly and without risking breaking things, requires not only vast and detailed knowledge of the system, but also knowing the context and decision around when the feature was added.

A long tenure engineer is the perfect fit for this. They know how to surgically remove the useless code. They remember all the hopes they had for the feature and when it failed to produce those results. They know well that code is not free, it's an ongoing cost to maintenance and complexity. This is not easy work to give to a junior. It's not even for senior engineers because they often lack the context and broad system knowledge. Only the long tenure staff engineer can confidently say, that feature failed, and I can remove it without breaking anything. If someone brings up sunken cost fallacy and how some users use the feature, only the long tenure staff has the background, respect, and authority to say, no this feature failed, it has 1% of the results we wanted, it's not worth the ongoing maintenance burden.

A staff engineer is supposed to a team multiplier through code. People often think of designs, refactoring, helpful tools, architecture discussions and documents. And it's true, these are all code related things that can multiply a team's efforts when done well.

But what better way to multiply a team's productivity than to reduce the size of the system? Think of how green field projects start. They feel so fast and progress is so quick. Then the features come, more and more, slowing down development with expanding system complexity and maintenance of code. How much faster and quicker can the team move after the staff engineer has reduced the system by 50%? It's like turning back the clock on an aging system. Suddenly it feels years younger, able to move quickly again, but this time with the right features in place from the experience of past years. And the team is happier, less useless code to wade through and understand, less strange bugs caused from code that supports a useless feature.

It's never actually useless code, it's the code that is _almost_ there. Then there's the whole political dance where one team still uses it because the replacement isn't quite the same and now you're solving some obscure use case and prioritizing the work gets done. Plus, it's likely the original author is still present so you have to placate them too.

I definitely take on the grim reaper role and make deletions happen. But it does take all the skills in the post.

We're going through this at my company currently. The best way to increase the capacity of a small team is to decrease their maintenance burden and the surface area they have to cover when implementing new features or changes to a complex system.

I don’t disagree that purging unneeded code has value. That being said, software systems are so complex, isn’t it better to just leave code alone if it’s not negatively impacting the business? How many orgs have a micro service architecture that is so well tested that you can delete code from one service and demonstrate that won’t cause a failure in some other service? I haven’t seen one yet.

Deleting code shouldn't be this hard, by logging how often a piece of code is being invoked in production one can start planning purges of unused dead code. It has to be done carefully but by no means it is so complex that it shouldn't be done.

Dead code is often a negative for maintenance, understanding and performance. With version control one can always go back and recover deleted code that proves to be useful later on.

There’s a cost to keeping the old code around, it’s just hard to see and quantify because it’s not acute (e.g. it doesn’t cause an outage).

At some point the cost of figuring out how to remove the code will be less than the cost of keeping it around.

Ways old, unused code can bite you in the ass:

* When new folks are onboarded, they should be spending a lot of time looking through the code base. Maybe on their own, maybe as part of other tasks, but they'll be reading this dead code multiple times.

* If the dead code is an entire method, and hasn't been used in a while, it probably doesn't work. What happens when someone calls that method now?

* Your CI/CD pipelines spend time building (and probably testing) dead code.

In some of the less decent companies that I've worked on, those supposedly janitorial tasks were pushed to junior engineers, while more senior ones were left to do be architecture astronauts.

Nothing ever could be done.

Searching code for usages of a piece of legacy code that you know is probably not used anywhere can be a good way for juniors to learn the code base, though.

For finding and deleting dead code anyone can get up and running by using something called tombstones, marking pieces of code as dead and then logging their invocations to see if they are really being used and if yes then how so.

It is much more powerful than simple static analysis and is setup as a continuous destruction platform alongside continuous integration.

See https://www.youtube.com/watch?v=29UXzfQWOhQ for more on this idea.

This sounds interesting, thanks for the link!

After recently joining a new company this was one of my big contributions, I went through the whole codebase and removed legacy/unused code, or merged code that was slightly different into a more general version.

Spent a good 2 weeks on this but the result was around 120 files changed and over 4000 lines removed, it felt great! And it was also an amazing opportunity to understand the codebase and ask the right questions to other team members when figuring out what was outdated and what wasn't.

Dear god. I hope you were asked to do this?

Ahahhahh.. Yes, I hope so too.

I'm sure it was great, but always beware to change two similar codes into a more generic one, when what was originally two served two different parts of the system.

In that case, what can happen is once the requirements change for one part of the system, now you must change a generic component in such a way as to not disturb the other part using the same component.

That way madness lie. (Of course, cut-n-paste programming is also madness inducing.)

>A thought I had recently is that staff/principal engineers who got their position by long tenure (not hired recently) should delete more code...

I still consider one of the highlights of my career to be a release of mature software where I was able to deliver the whole additional feature set on time, AND reduce the overall size of the codebase by about 15%. There was so much cruft from a long history of incremental development that there was duplicate logic, lack of single responsibility, and many bits of code that were 90% similar and could be made redundant with some basic abstraction.

It was glorious!

Wow! As someone who has one of these nebulous titles ("Software Architect"), this is one of the most accurate descriptions I've seen of what the job entails. My employer is nowhere near the size of Salesforce, so a lot of things scale down, but the basics are the same.

I would say the biggest challenge with roles like this is the loneliness and what seems like constant conflict. One needs to work hard to keep up personal relationships with product/sales/project management/etc to ensure that the level of trust is there to be able to have the hard conversations. It is really easy to forget that, and a lot of those other folks start to feel like you're always just shooting down their ideas.

> One needs to work hard to keep up personal relationships with product/sales/project management/etc to ensure that the level of trust is there to be able to have the hard conversations.

In the heat of the moment, on deadline with a bunch of bugs and yet multiple compliance hurdles to clear, with a short-staffed team and conflicting management priorities, it's pretty easy to lose sight of the common goal.

Microsoft has a nice cliche you can use when you catch yourself on the adversarial end of this type of conversation: "assume positive intent". It's written on most of the Redmond campus whiteboards. If we are all assuming positive intent, it can give us the right empathetic mindset with which to build this trust even when it seems difficult. I've found working with other teams through the hard conversations with compromise instead of "no" leads to better results. They're assuming positive intent as am I and it's OK. If I'm feeling something is adversarial I need to understand why.

I like to frame it as "not me versus you, us versus the problem" when things get too heated. Otherwise things degrade into ad hominem "Those goddamn X people", instead of "We're having a lot of problems getting our project to work with X"

I like this one, too. The moment the people are your adversary in a technical discussion, you have probably lost sight of the actual problem. Much of the time I find it's misaligned incentives or, as Amy wrote here, sometimes it's people working with obsolete context or old grudges. Reframing yourself first, and then the discussion, is very much the way to succeed.

My partner is a physician, and this is _exactly_ how they manage disagreements about treatment plans, almost by instinct. It is amazing how well this works at clearing things up.

What a great writeup. A lot of it sounds very familiar to my experience as a principal engineer at Amazon. There’s a certain point at which you realize the hardest software engineering problems for a large company are social problems, and that’s what you devote most of your energy to as a PE (or, I guess, a Staff Engineer).

Personally, I really didn’t have the stomach for the work, eventually; I have gone back to active programming, this time for scientific projects, and I am much happier.

Are you still working as a Principal Engineer ? Edit: just saw that you are at UW now.

So, you are now a researcher or scientists?

As someone who worked with Amy for years at Heroku, I can confirm that this is very accurate. Amazing write up, Amy. Damn.

Less years, but I concur.

>Everybody wants to hire senior engineers; nobody wants to make them.

Gold.

I have been struggling with this right now. I am not good enough nor experienced enough to figure out things on my own (to get to senior engineer). I feel I just need some guidance or push to get to the next level. But it's so discouraging to get no support from my management. And other folks around me also keep saying that depending on management to support me in my career growth is not a good idea. I feel it's ridiculous and distorts my image of the role of management. I just have to accept that good managers/management is hard to find.

Anyway, it's kinda good to know that it's not just my company/team that seems to have this issue.

Sounds like you need some meat (analogy aside of dietary preference) to sink your teeth into.

It's not a push because a push comes from behind. It's you grabbing onto something elevating and interesting because it's juicy.

A reasonable senior will recognize your talent, but a stagnant organization will simply extract your current value without pulling you up with that bite.

I know the analogy is bit inhumane, but the human part comes from those very human people who just want to solve problems.

Whether they be lower in pay/'rank' or higher, find the people who appreciate your strengths immediately. Don't ever be afraid to express your ideals in your capabilities and learning process.

In any organization from 10 to 10,000 you will find those folks who can take a genuine appreciation for your skillset and apply it.

The failure state should always be: "I'm departing to company Y, I wish you the best."

The success is an expression of your ideals.

The 'meat' of my argument is - at your level - don't look for the push, find the pull.

You've got this.

Edit: an -> and (Para. 5)

In my experience, management will generally not care about your career development and will prefer to hire a senior engineer from outside to avoid internal conflicts. The best way to progress your career is to train yourself up externally. For example, is there an open source project you would like to contribute to ? It is also worth to attend tech group meetings in your area of expertise. As everything is done remotely now, this has become even easier.

So here's a curious question that is perhaps incredibly naive and seemingly impolite, hence the throwaway, but I really don't know, which is why I'm asking.

Why must one be a senior engineer in tech before they can become a staff engineer? To me, this seems much more like a communication role. Moreover, the tech that is needed to know is at a high level. Is it really needed to know the technical finer details of systems when high level software architecture and communication seem to be 90% of the whole role?

Or put in a more blunt, albeit unrefined, way: why can't someone who specializes in communication become a staff engineer? I mean people like consultants. They get so much experience with communication and high level tech (some of them at least) that this seems to be what they'd be good at.

It takes a lot of experience to build the intuition to make the right technical tradeoffs, choose the right architecture, escalate staffing gaps, forecast timelines, rope in the right folks, etc. It’s not something you can learn in school, and there aren’t any shortcuts.

Your job isn‘t just to coordinate people, but help them reach consensus. If there‘s a tie, that decision is yours to make.

So:

1. You have to keep writing code to remain an effective tiebreaker 2. You still need to be a badass engineer in order to hire and keep exceptional talent who‘s looking up to you for mentorship. Respect and excellence are non-negotiable ingredients in a working meritocracy.

Sure you can also work outside of a meritocracy, but results and engineering happiness/productivity is usually subpar.

So that‘s exactl why the hunt is on for those unicorn engineers who can also socialize: It‘s proven to work.

There is a role like you're describing at some companies, often called a technical program/project manager. They're responsible for managing schedules, communication, etc.

The difference with a staff engineer is that TPMs are not expected to be technical experts, while a staff/principal engineer is. So they're not just routing information between stakeholders, they're also synthesizing and evaluating, partly responsible for the project going well and empowered to make changes they think necessary.

All of that takes a lot of experience in the trenches, having seen many projects succeed and fail. It also requires a lot of trust and respect, which takes experience to cultivate.

Interesting question. As someone else pointed out, TPM is a role more like what you’re describing.

For Staff+ engineers, communication and high level architecture are just the tip of the iceberg. You have to understand the problem domain and the solution domain, be able to come up with good technical solutions (which is generally a collaborative process), and sell others on these ideas. Then you have to follow through with implementing them yourself and leading others working with you on them. If you don’t have the technical depth to understand the details, you’re not going to succeed at that. If you haven’t already internalized a lot of the details through your past work, you’re not going to have enough time to both learn the technical details and do all the communication, implementation and other parts of your job.

I think a pretty good example of why this isn't work is visible in the LinkedIn inbox of nearly every software engineer (and probably other specialized people). People with an HR degree become tech recruiters and they often really suck at it on two fronts: spotting the right candidates and explaining the job. All they do is target as many people as possible and hope someone bites.

A really had thing for someone without actual work experience in the field is spotting the difference between: someone with the exact right experience, someone with the wrong experience but a valuable set of skills and personality to learn, someone bullshitting their resume and talking like they understand the tech stack and someone who seems to have the right experience but is actually a bad fit.

For a staff engineer it's critical to be able to spot knowledge gaps in teams, explain complex technical issues to business people, spot opportunities for the implementation of new tech and estimate the complexity of implementing it, etc. No matter how good you are at communicating, you will be unable to do this without actual understanding of the tech stack. Just explaining why last week it was ok to hire a react dev to fulfill an angular vacancy, but this week you really need a dev with super specific knowledge to fill an urgent need is hard to do without understanding those technologies.

I think you're too harsh on recruiters. They're paid less than developers and recruiting is a numbers game.

They can't afford to spend time to get super deep technically, since it's not worth it. And I imagine that those that do dive deep just get a technical job for less pressure and more money.

> They can't afford to spend time to get _super deep_ technically

Maybe you're just extremely lucky.

I'm already happy if I get a recruiter mail where the buzzwords aren't even used in a nonsensical way, e.g. just from two sentences I know the person hasn't ever written a single line of code or worked in a project using any of the 10 buzzword technologies.

Made up examples "using exciting database technology like mysql and mqtt", using "programming languages like SQL and XML"

> They're paid less than developers

Are we sure about that? I thought one of the most common compensation structures is they got a commission of approximately 10% of the first year's salary for each hired candidate, payable after the candidate has completed 3 months at the new job.

I hope the average recruiter places more than 10 developers per year.

The smart ones develop relationships with frequent job-hoppers. It turns almost into an agent-talent relationship.

They're usually employees. They only see a chunk of that.

That's about what the company gets. Most recruiters don't own their own companies.

Another thing to keep in mind is title inflation.

The role that most companies these days call “Senior Software Engineer” is actually an intermediate-level role (e.g., 5 years of experience). Staff is arguably where the actual senior-level roles start. This corresponds with the transition from being told what to do to being told to figure out what needs to be done and make it happen.

If you are a Senior anything and being told what to do, you're not really a Senior.

Maybe that works for you, if you're in the right place getting paid the right amount, but we do ourselves a disservice when we allow this to occur.

My personal understanding was: junior: lots of tactical guidance

medium: less tactical, lots of strategic guidance

senior: no tactical, very vague strategic (normally coming from business partners).

Maybe that's just me, though.

>senior: no tactical, very vague strategic (normally coming from business partners).

>Maybe that's just me, though.

That's just you. I agree that's probably closer to how the terminology should work, but it doesn't. I'm unaware of any large company where senior engineers are getting direction directly from business partners.

You're lucky if "senior" engineers have 3 years of experience.

I see it the same way. I've had to have the "why did you hire me if you're going to tell me how to do everything?" conversation with a recent manager.

This is my experience, too. For example, if I look at the title breakdown across the three SRE teams at my employer it's

Senior: 53%

Staff: 35%

Senior Staff: 12%

This is really good. It's one of the best descriptions I've seen of what being a staff engineer is like.

When I came into the role I was stunned at the amount of time that I ended up talking to people and just trying to figure out what is going on.

This article is incredibly well written and rings true to me and my experience being in a similar role / working with people in such roles.

I like the notion of articulating a career path in tech as of: 1) Learning the ropes 2) Being a producer / writing code / driving design decisions 3) Being a multiplier - working across different teams to unblock / amplify other peoples work

3) is pretty much communication centric, while 2) is deep knowledge of tech involved

There is a lot of nuance to that of course (which I think this article captures well within specific organization). I'm curious if this is something any one else has the same experience with

So glad the term "hopes and dreams" was used. That's exactly how I describe half my staff level role. I move mountains to enable and materialize "hopes and dreams", or I destroy them without a sliver of doubt, in equal parts. That means doing at least all the things the post describes.

But the other half was only lightly touched on. Staff is as much spearhead as it is rearguard. So that means caring for legacy codebases, aging tech, and anything else that's falling behind. It's search and rescue and postmortems after things have gone catastrophically wrong.

Great great write up!

I do not know this person at all but after reading this I think I’d enjoy being her coworker.

Right? She sounds so competent and practical, but also kind. Love it.

I worked with her (I was on the team she was initially hired onto at Heroku) and she is one of the best hires we ever made, hands down.

If you enjoyed this, check out https://staffeng.com/

It's composed of interviews with Staff-plus engineers as well as guides for reaching and succeeding at Staff-plus roles

I love this site. Thanks for sharing.

> You may be furthering existing misconceptions or old grudges that really should die off. Actively invest in meeting new folks as a counterbalance.

This one hits home. I've been at a rapidly growing org for nearly a decade and have grown a massive amount in turn. Things that I thought were the case sometimes ceased being so, literally, years ago.

Great write-up. Not sure I'd want the job. Sounds like she was a good fit for it.

I was a manager for most of my career, and spent most of that time pining for the fio- er, my days as a coder.

So, after I left my career position, I reverted back to a coder. It's been nice.

Damn really nice read! I wonder: is this what enterprise software engineering really feels like up there in the high ranks? I don't know if I would ever enjoy doing any of what she listed!

I had a similar reaction but there are a few archetypes that are very similar to the duties of an IC. She mentions “Deep Diver” and that’s similar to my current role. I enjoy having systems thrown onto my lap and becoming an expert in them as quickly as possible. Then the organization uses my analysis to decide what to do next: keep it or pivot to a different strategic solution in the long-term. Ideally my deep dives result in useful documentation, code improvements, or process automation to allow the future deep diver to come up to speed quicker than I did.

As a staff engineer, you have multiple responsibilities you need to cover, and you can delegate some of them.

The reality is that in most teams you can delegate the "Deep Diver" responsibility most of the time. "Context sharing" and "Coaching" not so much.

So yes, as some comments in other threads mention, your coding/hard tech contributions diminish with time.

I think it is. In my younger days I wanted to rise up so that I could work on really complex and challenging code. But now I'm there and I hardly code at all. Most of it is working across teams to figure out dependencies, blockers, security concerns in launching a new project. About a year in, I still don't like it much, nor do I feel like I'm good at it, even though I consider myself a good coder. I asked my manager for a transfer to a more code forward role and apparently they really don't exist at the high levels. There are some exceptions if you are a known expert in some field, but for a generalist in optimizing queries or caching patterns, there's usually someone cheaper and younger they can get to do the work.

> If we made the wrong bet, some blame is with product, but it’s also on you, and the manager probably won’t look great.

Why would any manager in his/her right mind choose to take any major risks while working for a big company? Maybe I'm looking at it wrong, but "looking not great" because you failed makes sure that next time you'll have to make the choice you'll always going to pick the safe bet. And safe bets aren't good for any technological company long-term.

I am with you on this. There is some positive incentive if the technical manager gets the product right maybe a strongly exceeds for a year. But if the product fails TLM gets equal share of negative press and a low rating which means greatly low bonus and bad reputation which will make hiring even more difficult. I don't see any reason why a TLM will take any major risks while working in a large company.

What is the salary incl bonus for a "staff engineer" at heroku? I find that titles are meaningless and in the end it also boils down to show me the money.

Trade C's like lunchroom fruit snacks before the VCs get here.

Heroku is now part of Salesforce. They don't appear have an official Staff level, but per [1] the levels above Senior are Lead at ~$287k (total comp), Principal at ~$378k, and Architect at ~$557k.

[1]: https://www.levels.fyi/?compare=Salesforce&track=Software%20...

US salaries are insane

Yes, although to be fair, that's on the high-end even for San Francisco. Total compensation in the range of $100-200k is much more common.

That's not right. Base compensation of 100k - 200k for engineers is more common, and even that is usually for up to "senior" engineers or so.

unfortunately those are low, bay area FAANG (incl bonus + stock):

    junior: 200-250
    mid junior : 250 - 300
    senior: 300-350
    staff: 450-500
    principal: 600-700
    director: 800 - million+
    vp: millions

Really good article. Context sharing is really powerful.

Your job as a staff engineer involves going into a cross-team meeting and explaining in technical detail a given piece of work, thanks to which your/another team re-estimates a given ticket from 5 points to 1 point. Those 4 points of difference are your hard work, done over the course of a 30-60 minute meeting.

This is probably what people mean by "force multiplier".

Great questions and answers. Admittedly, I can’t help but feel there aren’t very many pain points mentioned, which makes me ask what’s being left out.

Additionally, how often was coding actually involved in this role? I hear this claim often but in my experience it’s rare. One-line fixes don’t count. Specifically, what projects were you a code contributor to?

I'm going to guess quite infrequently based on this first sentence of how she spends her time.

> While I do write code from time-to-time, it’s only after I’ve delivered on my obligations on these four functions.

For me, it's either the core or foundational part of the most important projects to the company, or, one liners followed by a paragraph of comments in some tricky/obscure part of a codebase no one has looked at in a while.

>> How do you keep in touch with how things really work as you spend less time on hands-on development?

> As someone who still does write code occasionally, this is something I struggle with. Our codebase shifts underneath me constantly, in ways I often don’t see. It has been my goal for the year to make sure I say “I’m not sure, let me check” even when I feel confident.

This is not at all consistent with Staff Software Engineer at any of the several Big Tech Companies You've Definitely Heard Of I've worked at. It would be more of a Distinguished Engineer level sinecure[1]. Staff is usually expected to spend at least 50% of their time actually writing software.

[1] Not pejorative. If I were the relevant hiring manager I'd be happy to let someone like, say, Leslie Lamport, to pick a name out of the hat, spend as much or as little time as he felt like writing software just to have him around.

Isn’t Leslie Lamport a distinguished researcher, not a distinguished engineer? Research tracks are fairly different from engineering ones.

Definitely extremely consistent with Amazon’s Principal Engineers. Internal surveys on time spent coding support it.

This is also how we do Staff Engineering at GitLab: you're expected to spend most of your time writing code. What changes is how you go about doing that, how you communicate things, what responsibilities you have, the expectations of you, etc.

I really liked the perspectives on the four obligations every day.

Planning

Context sharing

Coaching & Culture

...

What is a staff engineer?

The level above lead/senior.

Not every org has them though depending on size, I’m a lead who does a lot of what she describes and there is no level beyond mine that isn’t pure management (which I don’t want if I’m honest).

When it comes down to it my role is a lot of ad-hoc dealing with bun fights before they come actual fires.

For the past years it seems Heroku has kinda stagnated. When was the last time they announced a new feature?

It seems all the talent is focused on Salesforce.

This is truly great! Work in an organization with ~200 sw engineers and some of it is pretty accurate to my role as well.

Solver/Deep diver FTW

All the way.

Reply on news.ycombinator.com