It’s funny how this would simply be an unacceptable answer in a root-cause-analysis. “I fucked up” basically just means “yea, someone else might make the same mistake, nothing to be done to prevent it”.
From my own life, I worked at a start up early in my career and dropped the entire production database. I was troubleshooting a production bug and had the prod DB open in one terminal window and my local DB open in another. I meant to drop my local DB and instead typed it into the prod terminal and that was that.
In the RCA we didn't just end at "I fucked up". The line of questioning went beyond that into what gaps allowed that fuck up to be possible which lead to action items around restricting access to production data to only those who need it as well as ensuring we had proper backups (thankfully, we did at the time) and ensuring we had runbooks on restoring data in the event that it happened again.
1) made you wait one minute before it would do anything to give you time to reflect on your life choices,
2) made sure to begin a transaction so you could roll it back before it was too late,
3) showed you the affected tables & rows,
4) made you type "FUCK MY SHIT UP" to confirm and
5) pinged everyone in our #DBA channel "<username> is about to fuck up production by running <query> affecting <rows> in <tables>. To stop them from doing something stupid, type STOP or NO" and aborted if anyone did so within the waiting period.
Anyone having to use FuckUpProduction was treated as a process failure and we did the whole RCA song and dance to determine why the normal testing/deployment process was insufficient. When FuckUpProduction was designed, some team members expressed discomfort at the use of profanity. This was considered a feature. We had a "swear jar" in the office that you had to put a dollar in if you were ever saved from accidentally running a query in prod either by FuckUpProduction or by the restrictive DBA accounts. The jar had enough money to buy a pizza before I left the company.
That is amazing, will inspire myself.
Like a reasonable answer at all times to this problem is things like: why did you feel like you had to work this way? Why was it even possible to do so? Why was no one else aware this was happening? Why did no one else feel empowered to halt the process if they did know? etc.
That sort of personal fuckup is just...never actually a personal fuckup, not when the collateral damage is so large. "I was logged into production as superuser" has the obvious follow up issues of "why was it easier to do that then have a correctly limited account?" (this one happens all the damn time - want a new account? Well here's 10 pages of justifications we need...also no versus "fix production now, just grab superuser from the password DB...)
EDIT: And I don't say this as (1) someone who hasn't done this and (2) someone who hasn't ever been using superuser to debug production.
I feel like every single one of us who has worked in this role has this same story, haha. Mine was iTerm split panes, but still. Over a decade ago and I still have my own little system I (now) adhere to ensure that never happens again.
[1] https://www.atlasobscura.com/articles/pointing-and-calling-j...
> In the RCA we didn't just end at "I fucked up".
You didn't end at "I fucked up" because "I fucked up" is not the root cause. It is not an acceptable answer to the question that root cause analysis is seeking to answer. Which doesn't mean you can't use the phrase to communicate, it's just not an acceptable answer.
> ...when we say "I won't make this mistake again", we're basically saying "Here was the problem: I was stupid once, and I'll never be stupid again, and neither will anyone who runs this business."
(The whole talk is worth watching, it's only 14 minutes long and it's very entertaining.)