Communicating to Users During Incidents | Hacker News Reader

Communicating to Users During Incidents

onlineornot.com

38 pointsrozenmd13 comments

Here's one more piece of advice earned in the field.

If you have a lot of users, and some have been affected differently than others, you may be tempted to send different messages to different sets of users, depending on who has been affected and how. You will not be trying to mislead them, but rather, will be trying to be as clear as possible in spelling out to them their specific, personal situation.

Beware!! People will post about your incident online, compare notes, and will be confused when their letters do not match!!

(This does not mean that you can't customize the response to the user, but you should be aware of this vector for confusion and must take strong steps to mitigate it.)

Mind if I add this advice to the article? You make a good point!

> As soon as you're aware of an incident, tell your users!

These days you'd be lucky to get any information about an incident on a status page let alone being communicated with.

The only information you get is all green checkboxes. We've all been there before!

If you work with government, the SLA's are so tight that this will never be feasible. Large SaaS and infrastructure providers consistently game uptime metrics to reduce breach related refunds and compensation.

For one major provider I saw them count as "service uptime is calculated by contacting the service once every minute of the day, if the service is contactable within any period of that 60 seconds, it is considered up". So if you healthcheck request typically responds in 200ms, but their internal infra falls over and it takes 59800ms instead - it's still considered "up", not even degraded.

This is stretched further with some providers to the point where they consider returning the header of an HTTP request as "response returned", before any body is sent. So they'll stand-up networking infrastructure when something falls over to hold millions of connections while they fix the backend to actually serve the content.. Eventually.

Often you'll find this isn't even contractually well-defined and is only given to you after an outage and you question their reporting. They expect you to take them to court if you disagree with this definition.

I worked for a startup where the lead/founding engineer would always panic and think of excuses and sneaky things to try to slither away from responsibility whenever there was an incident. Not only responsibility to the customer, but responsibility to our team. This is when I knew that I needed to start looking for another job.

I've always gotten pushback because I'm told that Sales won't be able to sell if there's publicly acknowledged downtime. I'm not sure what they think 99.95% in the SLA means, if not downtime occasionally. But, that's the reality. Customers apparently compare status pages, and pick the one that claims the least amount of downtime. (I bet you can fabricate benchmarks too. People love those.)

If I ever start my own company, I'm just going to make the Grafana instance public, perhaps with the Y axis scale removed. Then users can see "hmm, we're getting a high error rate", go look at the dashboard, and say "oh, everyone is getting a high error rate". The problem I always run into as a user of a service is "did I create this bug, or is it just something they're going to fix". 99% of the time, it's something the service provider is going to fix, and they're already aware. But because they don't communicate, I have to escalate and mitigate. That's half a day gone every time. It is infuriating. (The worst part is the time when it IS your fault, and you're used to making it the provider's fault. Resolution times can be pretty low when you're playing the undercommunications waiting game.)

I tend to monitor my vendors in great detail, so I'm usually aware of an outage before they are. Nobody that I pay for a plan has any way for me to make my monitoring available to their teams. I had a call with someone once about it (a popular build-your-website-from-a-git-repo service), and they said "oh, we'd let you into our Slack for 5 figures a year". No thanks! Give me a webhook and I can ping you within seconds of your global load balancer going down for the 10th time this year.

Anyway this has become a rant, but fuck undercommunication. It wastes my day, and I am already locked into a contract, so my business isn't going anywhere. Let's be honest now that you have my money!

That's a great rant!

I mainly monitor my vendors since I sell an uptime monitoring tool, and it's a form of dogfooding - didn't realise there were other folks that did it too.

at almost every place that I've worked at, it's been the task of the manager/ senior managers to keep the stake holders, general staff/ users at bay, to keep the techies happy and free of harassment during severity one incidents

and that includes making coffee, tea and bringing sandwiches/ pizzas

I need to work at better places. Our managers are the ones who schedule the "quick chats" to "see where we're at". In particularly stressful situations I've been blunt in saying we're not going to be able to solve the problem during this meeting so please let us get back to work.

I worked at a company that had our team in a monthlong rolling crisis born of bad management.

They wanted updates for the director who didn’t even understand the technical issues at least twice daily. Frequently, it was done over a teams channel so it was a rolling half hour text conversation/distraction trying to explain technical issues to someone who was bad at listening. By the end of it I was ready to sabotage the project just to end it.

explaining fairly complex stuff to non technical people and using diagrams (we called them crayon diagrams) can be an art form. Its a balancing act of showing something too complex vs too simple ie,showing one box and saying its broken(I've seen that happen).

imho, The best way to start is by asking the stakeholder what they know about the system and asking them stuff. this is good in different ways,

1) you can assess their level of technical abilities

2) assess how interested they really are

3)engage with them and create a personal relationship

and take it from there, using analogies can be extremely useful

heh

In my youth, fixing a bug on factory machinery that had broken down (downtime costing £10,000 ph) with senior management and CO standing behind me watching what I was doing....

..... I told them if they wanted to be useful, then to fuck off and make the tea because watching over my shoulder wasn't helping.

Being blunt and factual is always the way to go :-)

Reply on news.ycombinator.com