The tech world is rallying around a young developer who made a huge mistake
qz.com
qz.com
As a first-day developer, he shouldn't really have had the ability to make a mistake with any consequences beyond lost time.
On a tangent, I was paranoid enough in one position to have all my production terminal windows have a red background with yellow text. That brought the proper, serious attitude towards my work.
I liked the green / white text theme the most with blue being my second favorite to write code in so those got the ones I would use the most. I was never a lover of the old amber terminals and red is just plain painful to write on. I guess the more pain your terminal causes you, the more dangerous the environment.
It's hideous, of course. But it works very well for me, and I don't suppose I'd be averse, if interest becomes evident, to sharing it somewhere for others to benefit from and/or peruse in horrified fascination.
https://gist.github.com/anonymous/8ec128e88dac85d93ed58fb518...
Outside GNU coreutils, the only dependencies are mysql(1) (of course) and jq(1), which is more or less "awk for JSON" and which I strongly recommend to anyone who regularly deals with that file format.
Share and Enjoy!
[0] https://biology.stackexchange.com/questions/3871/which-shade...
I was at a site when a truck hit the power main and the UPS (diesel if I recall) for the server room failed to work. The new people were in the "It shouldn't take too long to reboot the servers" where the experienced people were "this is going to be a very long day". The later group was rewarded with a 36 hour day.
export PS1=$(tput bold; tput setaf 1;)'(production)'$(tput sgr0)' \u@\h: \W $ '
I had a job where we would often sudo as other users and ssh into other boxes. I kept trying and trying to find a way to change the terminal when sudoed as someone else because occasionally I would execute something in the first terminal I could grab. At the time I couldn't find anything or think of a way.
Indeed. To capitalize on a meme in current circulation, "One does not simply 'accidentally destroy' a production database."
When things like this happen, they happen because someone who almost certainly did, in fact, know better -- at some point made a conscious decision to allow them to happen, by consciously cutting corners in certain matters of due diligence that we all know about. Things like backup systems, database roles, and hiring the right people to set up and maintain these things.
Not because of something a junior dev did.
Better yet, prod should be in a different "cloud universe" (root account, etc) from dev.
> The company/CTO made the huge mistakes.
Absolutely.
It doesn't make any sense that anyone would put a junior on his first day even close to a production environment, much less using a tool that can delete a database that btw you can't restore from a backup.
IMO this is either a false story, or this was done on purpose.
What if the company wanted to get rid of some data that would put the company in a difficult position in front of its shareholders or even the law? What if the lost data could prove that the CTO had made some other much bigger mistake that allowed hackers to get private information?
We will never know of course.
Never ascribe to malice that which can be explained by incompetence.
Silicon Valley and Veep are both disturbingly prescient.
A rule of thumb is that the really ridiculous stuff in Silicon Valley like the "three commas" or the "blood boy", actually happened.
One of the first things (so a couple days in, not exactly day one) the CTO had me do was "map" their server room (e.g. figure out where data and power cables were going to and coming from, physically and virtually). While working my way through the back of a massive UPS rack, I stepped on one power cord that happened to traverse over another 10 or 15 cords. I did it in such a way that all of those cords were pulled ever so slightly from their UPS connections. I didn't even realize it until one of the staff IT guys came in the room a few minutes later and asked me what happened. Turns out those cords powered the 300+ Citrix thin clients in use by the national call center downstairs.
I thought for sure I was in huge trouble, but everything came back online fine, the staff IT guys poked some fun at me and the CTO helped turn it in to a valuable lesson. I learned a lot of great stuff during that interneship and to this day, 15 or so years later, I still think of that experience when doing risky work (or when I fuck it up).
It's still amazing how badly that company is being managed from a technical perspective.
I see posts of this nature all the time on r/csquestions. This one just happened to go viral.
Lulz, getting one over on the Internet, being able to fabricate a plausible story and see people all over the world reacting to it. Some people get off on weird stuff.
See -- Programmer Automates His Job Away for 6 years and then forgets how to program.
I don't think any individual aspect of the story is unbelievable, but when taken as a whole it stretches credulity.
- It's a prank by the OP. He posted a story out of his mind for whatever reason.
- It was all manipulated by a different employee of the company. He (by accident) broke the production db and decided to blame the junior dev: Most junior devs will just believe you (out of fear and inexperience) when you say to them that they broke the production.
The 2nd one seems more possible to me.
When I'm fully in control I engineer things in a way that make disasters unlikely and easily recoverable. But that is not always possible. I tend to avoid touching any production DB but again that is not always possible.
I always believed I have a developer mindset more than an operational mindset and worked more in roles compatible with the former than the larger. When looking at the devops culture I always wonder whether the lessons of strict control of production environments are getting enough attention. There is value is closing the gap between the two world but please leave a gap.
(Note, this was many years ago and mildly traumatic, so some details might be fuzzy)
In 2001 I was working for one of the larger internet server co-location companies in the US as a dedicated customer support engineer (remote hands) for several large clients.
One of my clients requested that I perform a memory upgrade on their Sun 420R servers at 4am (on my birthday none the less).
So I come in. Confirm with the client they are ready to be shut down, confirm with the NOC that the servers are coming down. The client remotely shut down the servers in question but they didn't power off (missed clue). One by one I power them down, roll them out of the rack, insert memory.
This was a pretty routine request, so it went smoothly - so I had thought.
Switch the console to the database server, boot. The kernel starts without issue, then a slew of error messages. Too many to read.
The Sun 420R's were known to have memory riser issues, so I power off reseat the riser, and all the memory. Same problem.
I take out the new memory. Same problem.
Before I could troubleshoot the client guesses it's a hardware issue, suggests we bring up the other systems and the cold-backup systems to compensate.
Nothing comes up.
Not
One
System
So at this point the client starts to panic over the phone. They hang up on me.
So I pull out my laptop and switch to a serial console on one of the servers in attempt to troubleshoot the error messages.
All the error messages are Bourne Shell (sh, not bash) errors.
Realizing this doesn't appear to be a hardware issue, I boot from a recovery cdrom, and start inspecting the init system. Everything looked great. So I tried to run a script, and then it became obvious - everything LOOKED too great.
All the init files had been reformatted. It said as much in a comment. A certain very well known perl developer (at the time) who developed the site for the client had found the formatting of the init scripts atrocious and unreadable, so took it upon himself to reformat every init script on every server. The problem is, bourne shell didn't like some of the formatting changes - mainly space between variable, equals sign, and value declarations. Had he tested his changes, it would have been obvious what the problem was.
Testing my theory I fixed the formatting of a single script, and it immediately works without error. Problem found, now I need to fix all the scripts on all servers.
I call the client. No answer.
I call the client again. Answer. "Did you fix our servers yet?" "No, I found..." phone disconnects.
I call again, no answer.
I call again, no answer.
I call the NOC, asking they try to inform the client of the status.
So I proceed to start fixing the first server. It took an hour, but the server came up. The second server came up a little faster, and I proceeded to work on the third and fourth.
Meanwhile, I find out the client had been calling around escalating to senior management at my employer petitioning that they fire me and put someone competent on the job.
I find this out as one of the analysts comes up from the NOC and tells me I'm relieved, and was asked to go home with no further information. So I explain what had happened and what I was doing to fix the issue.
The next day I am called into the office. Told I was fired for my obvious screwups, and that thankfully the NOC analyst had been there to save them from losing such a large client.
Happy birthday.
Without question, there were things I could have done better in that situation, but could not have predicted in a million years that someone would reformat init files without testing them.
I really don't want you to think I'm criticizing your actions - you were a true hero to the customer in a crazy situation. I think its truly unfortunate that they terminated you for this. Best of luck!