Hacker News Reader

Comment by patio11 | Hacker News Reader

Hey guys -- the most resource-intensive part of the infrastructure (the game master boxes) is currently overwhelmed by your enthusiasm. We're spinning up more boxes for you, but it will take a few minutes.

View on HN

I BLAME RAILS.

Thomas and I have a running gag about this. Rails is holding up admirably. You can still actually see the website. Our problem is presently that one player implies 1+ ruby (not Rails) processes running bots and doing orchestration for their levels, and that has pegged the CPU on all of our GM boxes and also, possibly, saturated their network links as they hit the venue with... I don't actually know how many orders have been placed already but I'm going to wager a guess that it's "a lot."

I so want to read a postmortem of the launch issues. It's one thing when a corporate project with crazy requirements goes down (almost expected these days :( ). Completely different when a service started/run by known tech people with their own schedule breaks on launch. I'd be really interested in what testing was done and why it didn't match reality. (not a jab/criticism, just something to actually learn from)

Meh? It's a play-money stock exchange which processed over 200k orders since a few seconds ago. We're in the process of finding the parts of the plumbing that weren't sized right for hundreds of simultaneous players and tens of thousands of simultaneous actors (each bot is approximately as taxing on the infra as a human -- more, at the moment, since many folks can't read the instructions to start hitting us with orders and the bots already know how to do that).

Don't underestimate how valuable "meh" post mortems are. I for one, would love to see your usual writing ability applied to one.

Weeks from now when this is all a funny anecdote of course...

I confess that I would love to see Patrick's usual writing style and ability applied to almost anything. I'm not sure why, but he just seems to be really ... genuine? Personable? It's like I'm listening to a friend talk -- a friend who happens to either be an expert, or have really interesting things to say.

I want to say that it's bizarre, but I think it's mainly that he writes very well.

I wondered if Patrick wrote "The StarFighter Handshake", the friendly guidelines that take the place of Terms And Conditions when you signup. It made the usual legalese (ie don't DDOS our site, don't hack into personal user data) feel like shared values of a club you were about to join. That had a hugely positive reaction on me when I saw & read it, and I'm not even entirely sure why I was so impressed. [Kudos, whoever wrote it.]

Did you guys consider using Elixir/Phoenix? Seems like it would have been a good fit (in terms of scalability) for what you guys have built.

> Rails is holding up admirably. You can still actually see the website.

Site is down now... (at least in Europe)

Site has been mostly down for me in the US over the last ten minutes.

Give it time, for me it thought about it a few minutes, and then rendered :)

> Rails is holding up admirably.

... except that you have to spin up more boxes.

I think you have a different definition of "admirably" than most of us :-)

Follow Twitter's lead and switch to a JVM-based solution when you get a chance, you'll be shocked.

We have one Rails box w/ ~2 gigs of RAM which has been sitting pretty since the golang process also on the box stopped hogging 100% of the available file descriptors. Our scaling issue is with ruby processes, of which we have several thousand active at the moment. Each process runs one wee little universe, filled with wee little traders trading wee little stocks, for the benefit of a single player playing a single level.

These aren't a JVM language (or Golang, for that matter) out of expediency: I had to write the economic simulation and the trading algorithms and get it done in about ~2 weeks, so I wrote them in my best language, Ruby.

Curious--is this something with which JRuby could help?

I'm a fan of the JVM, but it doesn't remove the need to be prepared to scale up by adding more machines. That said, it's great at concurrency, so it's easier to harness the full capability of a machine than with other platforms.

Even if I didn't need multiple machines for throughput I'd probably still want them for redundancy (redundancy of machine hardware as well as redundancy across availability zones). (Note: didn't vote on your post)

I thought your stack was mostly go?

It currently appears to be mostly stop.

It's mostly Go except where it isn't.

I suspect that was irony...

In a blog post some months ago [1], Patrick said that some components do use Rails.

[1] http://www.kalzumeus.com/2015/08/20/designing-and-building-s...

I know using Rails isn't as popular here as it once was, but in my experience, if you cache correctly (like you should), it's trivial to scale.

Modestly less trivial if you happen to put your chat server on the box with Rails because Rails is certainly not going to fall over and your chat server attempts to open several thousand websockets on a box with a not-too-generous number of open file descriptors, it appears.

"Make sure you've turned up the file descriptors" is the single-process socket server equivalent of "Turn off HTTP KeepAlive" for Apache sites. :)

(Yeah, I've had outage because of this too.)

Surely you meant "Turn on HTTP pipelining"? Or is there something wrong with Apache's pipelining implementation?

As I understand, HTTP pipelining is basically a no-go in the real world. Broken implementations, plus performance implications since responses must go in order.

Edit: Here's a FF discussion on it: https://bugzilla.mozilla.org/show_bug.cgi?id=264354 -

I misremembered a bit; it was Turn Off Apache Keepalive: https://hn.algolia.com/?query=patio11%20apache%20keepalive&s...

I feel like a chat server is something best served by something not in your stack (especially Rails). I surprised you chose to do it in house. How about using https://pusher.com/ or https://www.firebase.com/ ?

Perhaps those lack useful vulnerabilities?

Why in this day and age of cheap and easy servers would you ever have more than one service type running on a server in production? It's trivial to break things up nowadays

Looks like it's actually Go based on one of the JSON responses I found in the web page:

    {
      ok: false,
      error: "strconv.ParseInt: parsing "NaN": invalid syntax"
    }

Congrats on having the problems you want to have. :)

These are definitely good problems to have!

I hope you get this fixed soon. It can be quite stressful having your exposure period potentially damaged because your software or infrastructure can't handle the sudden influx.

Stick to deep breathes and think it through :)

I'm pretty calm at the moment, particularly now that the servers seem to be cooling down a bit.

Startup culture says Launch Days Are Really Important. This is not our first rodeo. We know 0.01% of the people who are going to play this game are even aware it exists at the moment. We're pleased as punch that y'all are playing and some of you are progressing in it, but long-term, a few teething problems on launch day is a no-op.

Reply on news.ycombinator.com