HNHacker News
TopNewBestAskShowJobs

TheSwordsman

1,839 karma · joined June 12, 2012

[ my public key: https://keybase.io/theckman; my proof: https://keybase.io/theckman/sigs/sdBoJTtBwPcwi1tpmyge7iA4vooZ1ZmTK_huiHJGylc ]
submissionscomments
TheSwordsman··on Ask HN: What is it like to work on pager duty?
The first-responders at my company are considered higher up on the totem pole. They keep the ship afloat, while others get to sleep blissfully ignorant to the latency that's causing replication to shit itself...

If there is an issue that another team is responsible for, we open up tickets on that team to address the issue. We take our jobs seriously, and make sure things get taken care of. It's not an easy job, but it's everyone's responsibility.

You also need to have the bosses on board to ease the pain of on-call. They need to allow you to come in late, leave early, or take the day off. They also need to be mindful of on-call fatigue and shifting the pager around to mitigate that.

Maybe we should stop treating first-responders as the low people on the totem pole, and treat them as everyone else. They keep the wheels running at night, and you may not be getting a paycheck if there wasn't someone there at 4AM to answer the phone.

TheSwordsman··on Ask HN: What is it like to work on pager duty?
This seems like a very naive response. We run on hardware that's lifetime is quantified not whether it will fail, but when it will fail. You don't know when that is, or how it will fail. The node could completely go away, or degrade enough that it begins to impact performance.

We also run persistent systems across the WAN. And, unfortunately, some of these things require the state to be maintained.

You can't just design these systems to be "better". There are often things outside of your control.

Based on your response, you seem to be the type of person causing pain for those with a pager.

Also, I'm sure the company that can make the Internet work every time, all the time, will make a killing.

TheSwordsman··on Ask HN: What is it like to work on pager duty?
To echo akg, I've also been in experiences where the boss and team make the experience great. It's great if you like working on small tasks that are truly dynamic.

Sometimes, getting paged allows me to get my head clear of your normal responsibilities. I kind of use the pages as a nice refresher for my regular work. However, personal preference.

To comment on one of akg's points, the length of the schedule rotation can actually be a problem. Longer than 6-8, and you are then very detatched from the operational issues your team is having. Make sure you link up with the previous primary to get the run down on anything (and maybe even the person who was primary before her).

You definitely want to get more familiar with the team you will be rotating with, but:

1: Do not fear escalating to secondary / asking for help when you need it. While getting paged sucks, every team I've been on this has been one of the biggest guidelines. We would rather someone ask for help sooner, than do something they are unsure of / delay fixing something.

2 (More like an extension of akg's #6): This is something to clear with your boss, but when I'm on call I also try to put the engineering effort in to fix alarms that are less than ideal. If I get paged at 3AM for some bullshit reason, I will spend the time the next day to make it alert when there is something actionable to be done.

If you are at a startup it's probably unlikely you'll be reimbursed for your cell phone (receiving calls). However, they should have provisions for mobile internet. Using your personal cell phone for work-related activities could have interesting legal implications. If the company is getting investigated, your personal phone could (in theory) be included as work-related things have gone through it.

I've been on pager duty for a few years now. I've no regrets. However, I'm sure there's one day in my life where I'll be over it.

Best advice I can give while on call: keep calm and have fun.

TheSwordsman··on $2.25 Lyft line rides in SF
Pretty neat idea. However, more glad that Lyft finally has the ability to split fare.
TheSwordsman··on Day in the Life of a Google Manager
Sometimes just reading mindless things at the end of the day is the perfect way to end it.

I find myself using the evenings to catch up on email, because I can take my time and do it in between other things. I'm sure once family life becomes more prevelant I'll change my tune.

TheSwordsman··on Docker and the PID 1 zombie reaping problem
And people who blindly push systemd fail to realize that we've alreqady solved these problems. A more complex system doesn't solve it.

So instead of a 350 line Python script, you think systemd is better? I disagree.

TheSwordsman··on Don't Make Visitors Make Accounts
Nevermind, answered my own question.
TheSwordsman··on Don't Make Visitors Make Accounts
I gotta ask, how many systems/websites have you actually maintained in a production environment?

I like people offering help to others, but I read your article before you updated this... and I immediately wrote you off because of that.

It's great to offer advice, but can't seem to find any indication of what your experience is.

TheSwordsman··on Don't Make Visitors Make Accounts
Who is BenBurwell, and why do I trust his advice?
TheSwordsman··on We have broken HTTP
Agreed. This guy needs to take a break from the Internet for a few years.
TheSwordsman··on A positive leap second will be introduced at the end of June 2015
Google wrote a blog post about this a few years ago:

- http://googleblog.blogspot.com/2011/09/time-technology-and-l...

TheSwordsman··on Rapid DHCP: Or, how do Macs get on the network so fast? (2011)
You do realize you're partaking in unwarranted Linux adoration, yes? Quite literal situation of pot calling the kettle black.

If this were truly about the merits and capabilities of Linux as a kernel and whichever userspace utilies you slap on, there would be no self-importance in your comments. However, it's more about showing off your accomplishments and trying to make yourself feel and seem superior.

I applaud you on your boot time. Sadly, it's not really applicable anywhere beyond the confines of your workstation.

Systemd is great and all for desktops, but I like keeping that complexity and invasiveness away from my servers.

TheSwordsman··on Rapid DHCP: Or, how do Macs get on the network so fast? (2011)
I was trying to avoid sounding pompous, while calling out someone else for being pompous...

It's a delicate situation!

TheSwordsman··on Rapid DHCP: Or, how do Macs get on the network so fast? (2011)
No, let's not kid ourselves. You're being downvoted for being a bit pompous.
TheSwordsman··on About the OS X Yosemite v10.10.1 Update
Same problem. Upgraded immediately hoping it would fix this.
TheSwordsman··on Shall we fork Debian?
Which makes using any useful features of the project impossible.
TheSwordsman··on Shall we fork Debian?
As someone who manages a large Debuntu fleet, please don't speak on my behalf. If I had the opportunity to vote for systemd it would have been 'no'.
TheSwordsman··on Shall we fork Debian?
Experience from a few years back. Bug was fixed and all, but left a bad taste in my mouth.
TheSwordsman··on Shall we fork Debian?
And if that service file screws up how do you troubleshoot it? You end up diving in to the source code of systemd, versus fixing a bug in your script.

And don't get me started on binary log files.

As was said before, buster, your attitude is the problem. It's the same as the systemd developers and those within the community.

SysVInit has served me thus far with no issues.

TheSwordsman··on Pinboard 2014 Expenses
Sounds like Fremont 1...
TheSwordsman··on Bits from the Debian GNU/Hurd porters
I'm probably being over-pedantic here, but you know that technically the kernel is the OS, right? The userspace tools are not the operating system. So Linux distributions are Linux operating systems, with tools from the GNU project.

Back to your sentiment, yeah it makes sense for all things GNU to be able to work with each other. That's one of the biggest benefits of this Debian project.

TheSwordsman··on Google Cloud DNS
That and DNSSEC support. If they had DNSSEC support, I could see my organization moving to them.
TheSwordsman··on HipChat Will Grant Employers Access To 1-to-1 Chat History
They definitely aren't rare. We have the same pain where I work. We just end up having a limited number of HC admins and make them do all our API requests.

It'd be nice if you could take a conversation off the record. Is it inconceivable that HipChat may be used for human resource like discussions? I see some risks with this, and not sure I agree.

However, I do understand why they would do this. Bummer.

TheSwordsman··on DigitalOcean Raises $37.2M From Andreessen Horowitz to Take on AWS
MIPSpace is stupid if they think you can get SWIP from DO. This is not DO's policy, but the RIR. I seem to remember that for ARIN you need a /30 or larger of consecutive IP address space to qualify for SWIP. This may even be a /28.

So while DO is saying no, I'm wondering if it's because they don't offer the space required to SWIP the ranges. In addition to not offering the size, it would incur additional administrative overhead.

Can't blame them.

TheSwordsman··on Redesigned Python.org
Why does the site return gzipped content for curl? I'm fairly certain curl doesn't provide the 'Accept-Encoding' header by default.
TheSwordsman··on CandySwipe Open Letter to King regarding trademark
I'm guessing the cost of actually doing this is more than he has. And even if he wins, I'm sure King.com will spend months or years trying to appeal the ruling which will cost him more money.
TheSwordsman··on [dead]
I've looked through this. It looks to be a sanitized version of their database with very old information. The reason I say sanitized because there look to be little or no credit cards there, and the only ones that look like CC numbers are '4111111111111111'.

My guess is that this is an old development DB that was left on a server that may have been forgotten about.

TheSwordsman··on Github was down
GitHub has a service that is meant to be highly available. What are the chances that 1/4th of the engineering effort would be placed in to a local solution?

Downtime is inevitable, and sometimes the thing that fails will cause huge problems and it's almost entirely unavoidable.

I'd much rather trust their team, which they've built to design the best service possible, to handle the infrastructure than having to use an internal team to design a competing product (albeit internal) with the same uptime requirements.

TheSwordsman··on Github was down
I wonder if they were using DNSSEC on their internal DNS...

http://www.dynstatus.com/dynect-dns/5628/possible-dnssec-iss...

TheSwordsman··on Namecheap now offer .io domains
Probably NetworkSolutions...
← PreviousPage 6 of 8Next →