HNHacker News
TopNewBestAskShowJobs

jssjr

243 karma · joined June 29, 2013

VP Infra @ Fly.io building cool tech for smart people with an incredible team. Give me a shout at scott@fly.io if you want to connect.

Previously VP Infra @ GitHub

submissionscomments
jssjr··on Fly.io Postgres cluster down for 3 days, no word from them about it
I appreciate the honest feedback. We could have done better communicating about the problem. We've been marking single host failures in the dashboard for affected users and using our status page to reflect things like platform and regional issues, but there's clearly a spot in the middle where the status we're communicating and actual user experience don't line up.

We've been adding a ton more hardware lately to stay ahead of capacity issues and as you would expect this means the volume of hardware-shaped failures has increased even though the overall failure probability has decreased. There's more we can do to help users avoid these issues, there's more we can do to speed up recovery, and there's more we can do to let you know when you're impacted.

All this feedback matters. We hear it even when we drop the ball communicating.

jssjr··on DDoS Attack Against Dyn Managed DNS
Another status update from GitHub: "We have migrated to an unaffected DNS provider. Some users may experience problems with cached results as the change propagates."

We're maintaining yellow status for the foreseeable future while the changes to our NS records propagate. If you have the ability to flush caches for your resolver, this may help restore access.

Latest status message: https://twitter.com/githubstatus/status/789565863649304576

jssjr··on Massive Dyn DNS outage
I'm a GitHub employee and want to let everyone know we're aware of the problems this incident is causing and are actively working to mitigate the impact.

"A global event is affecting an upstream DNS provider. GitHub services may be intermittently available at this time." is the content from our latest status update on Twitter (https://twitter.com/githubstatus/status/789452827269664769). Reposted here since some people are having problems resolving Twitter domains as well.

[x-posted on https://news.ycombinator.com/item?id=12759697 as well]

jssjr··on DDoS Attack Against Dyn Managed DNS
I'm a GitHub employee and want to let everyone know we're aware of the problems this incident is causing and are actively working to mitigate the impact.

"A global event is affecting an upstream DNS provider. GitHub services may be intermittently available at this time." is the content from our latest status update on Twitter (https://twitter.com/githubstatus/status/789452827269664769). Reposted here since some people are having problems resolving Twitter domains as well.

jssjr··on The GitHub Load Balancer
Sure. Feel free to call it anycast then. I usually hear anycast routing used in the context of achieving failover or routing flows to the closest server/POP, but there is probably a more formal definition in an RFC that I'll be pointed to shortly. =)

We are using BGP to advertise prefixes for GLB inside the data center to route flows to the directors. In our case all of the nodes are not on the same subnet (or at least not guaranteed to be) which is one of the reasons why we chose to avoid solutions requiring multicast. I expect Joe and Theo will get into more details about that in a future post though.

jssjr··on The GitHub Load Balancer
Anycast usually implies traffic will be directed to the nearest node advertising that prefix. The GLB directors leverage ECMP which provides the ability to balance flows across many available paths.
jssjr··on The GitHub Load Balancer
A "whole" piece of software would require you to have made the same data center design decisions we've made at GitHub. While some of our choices are opinionated, I think you'll find the GLB architecture adheres to the unix philosophy of individual components each doing one thing well.

Either way, I hope the upcoming engineering-focused posts are interesting and informative! Developing GLB was a challenging engineering project and if open-sourcing it means other companies can benefit from our work and spend more time developing their products, then I'll consider that a success.

jssjr··on Introducing unlimited private repositories
I think you're referring to the GitHub Engineering blog post [1] about our git storage tier. We [2] store your code on at least 3 servers, which is an improvement in many ways from our previous storage architecture. There are a lot of servers [3] powering things but not the millions it would require to give every customer three dedicated machines. Developing efficient solutions to problems is a requirement (and a fun challenge!) for anything at GitHub's scale.

[1] http://githubengineering.com/introducing-dgit/

[2] I'm a GitHubber. https://github.com/jssjr

[3] https://twitter.com/GitHubEng/status/730429227896463360

jssjr··on How Fastly coded their own routing layer for scaling CDN
This is really great work. Do you have any plans to open source some (or all) of the code behind Silverton?
jssjr··on GitHub's Metal Cloud
GitHub's physical infrastructure team doesn't dictate what technologies our engineers can run on our hardware. We are interested in providing reliable server resources in a easily consumable manner. If someone wants to provision hardware to run docker containers or similar, that's great!

We may eventually offer higher order infrastructure or platform services internally, but it's not our current focus.

jssjr··on Atom 1.0
It's here at CodeConf!

https://gist.github.com/jssjr/018717ddcb81b76e1829#file-img_...

jssjr··on Brubeck – A statsd-compatible metrics aggregator
We use collectd extensively and it is wonderful software. Brubeck and collectd do very different jobs.
jssjr··on Brubeck – A statsd-compatible metrics aggregator
We did not try out statsite, but it also looks great! Brubeck has been under development internally since October 2012. We're really excited to be able to share it with everyone.
jssjr··on Brubeck – A statsd-compatible metrics aggregator
Indeed!
jssjr··on Rearchitecting GitHub Pages
This sounds like a great idea for a future blog post. Thanks for the suggestion!
jssjr··on GitHub Launches an Engineering Blog
There are two posts on the blog right now:

Workload Analysis with MySQL's Performance Schema

http://githubengineering.com/using-mysql-performance-schema-...

Browser Monitoring for GitHub.com

http://githubengineering.com/browser-monitoring-for-github-c...

jssjr··on Uber's playbook for sabotaging Lyft
http://gocurb.com/ has been doing exactly that for many years.