Hacker News Reader

> People with real experience will have encountered this problem and know that you have to spend time investigating to find the root cause of the backpressure, otherwise you could add capacity to random things without reducing the queue occupancy. Are your queue consumers getting throttled by some downstream service? Are your workers CPU bound? The "right" answer here is basically "we should figure out why the queue is filling up before changing anything", but very few candidates answer that way.

This is the opposite of what "people with real experience" do, at least in production-down situations. The first step is to mitigate, the second step is to root cause. If there's some no-brainer step that has a chance of alleviating the issue while you root-cause, and is unlikely to make things worse, you should take it.

Your point is good, but in making it, you sound like an adversarial interviewer.

You and OP are actually agreeing with "the correct answer is it depends and now let's discuss context". This echoes my experience as interviewer too. It's a red flag when the candidate responds with "the correct answer". That's what OP is calling out.

I'm replying here because I get the impression you're looking for "the right answer" as you see it: "the first step is to mitigate then do root cause". You're right! But it also could be too adversarial.

> Your point is good, but in making it, you sound like an adversarial interviewer.

Most interviewers are adversarial.

Let's be honest - if an interviewer wants you to pass a system design interview, they'll make it work. I see this with particular candidates all the time. If we want the person to make it through - we'll let them get through. If we don't want them to get through - no amount of correct and behaviorally appropriate answers are gonna make them get through.

Pretty much this. You can have experience and also perform well in a system design interview. However, passing some of these rounds involves chance. Some interviewers are looking for specific answers. For example, some interviewers are looking for you to be knowledgeable about recent papers specific to the heavy-hitters problem. Or the two ways to best support lynchpin objects. It’s not so much as dropping well-known building blocks to construct a system but rather the interviewer asking you to design a system requiring specific characteristics you might not have any specialization in building.

This might be true of SRE roles, but not really true for most engineers, particularly not where I currently work. I wasn't completely clear but what I'd be asking for here is how to predict and design for bottlenecks, not how to mitigate active disasters.

Also making a queue bigger is a bad reflexive response to queues being full. I was very involved in the SEV review (postmortem) process at Facebook and witnessed lots of cases where bad situations were made much worse by misguided hasty responses.

do you work at google? most engineering roles ive been in engineers absolutely own their systems and take on call rotations.

I agree though, hasty response is the mark of a junior eng

I do, but I don't interview SREs. "Production down" situations are hopefully rare, so the purpose of the interview is to understand if the engineer can plan ahead by adding monitoring, logging, and knobs, and to know how to design for scale. I don't really ask about what to do in a fire because many people don't have that experience.

> Also making a queue bigger is a bad reflexive response to queues being full.

Depends on the function of the queue. If it's a dead-letter queue, there's literally no downside besides a likely-trivial amount of cost. If the problem is upstream of the queue, and the queue is actively feeding well-functioning consumers, yeah, it could make things way worse. This is where being an experienced engineer comes in. Also having 2-person approval like you would for any code changes. Point still stands that you should take low-risk actions to mitigate if they're available to you before root-causing.

It feels cool to me that my nontraditional degree for software (ChemE) drilled this into us and now it comes up in my current job: Rate of input - rate of output = rate of accumulation

It’d be fine to just up the queue capacity if the input is just inconsistent/spikey but averages to the input, but under steady state (or steadily increasing growth) higher input, you must increase the output rate or you will outgrow whatever capacity the system has

What do you expect them to say instead?

I take your word for it that they’re trying to trick you or something - it’s not my story. But it kinda comes off like you’re denigrating people for not knowing something.

If they bring up graphql, I expect them to say something like "none of the advantages of graphql are applicable in this case, so it may make sense to use if it's already well supported. All else being equal it's probably not a good fit"

The key is that the response should reflect the actual design we're discussing, it should be more specific than something that could be said in any interview.

In an interview it is a negative to not know something. It's a bigger negative to not know something and try to hide that, or to be ignorant of the ignorance. "I don't know" is a fine answer in an interview. "I don't know, here's where I would look" is better