HNHacker News
TopNewBestAskShowJobs

jordanthoms

2,885 karma · joined March 14, 2012

jordan@thoms.nz

Co-founder, Kami - kamiapp.com

submissionscomments
jordanthoms··on Patching Rails Performance
Puma's currently the recommended one, but I think that's actually a bad recommendation.

Puma is great if you have a Ruby implementation without a GIL (e.g. JRuby, which it was built for), but it's not so good on MRI, because you have to use clustered mode, which runs multiple processes. What they don't tell you is that when running clustered mode, the requests are randomly distributed to each process, and then the requests queued at that level. Each process can run multiple threads each processing a request simultaneously, but because of the GIL anything that's computationally heavy will cause the other requests in that process to run slowly.

That means that it's possible for one process to become backed up with requests, while the other processes on the dyno are sitting idle. And it's actually not just possible, but likely when under reasonable load.

So there is actually no deployment configuration of Puma which makes sense under MRI, at least without a reverse proxy on the server handling queuing:

- Single process (no clusters), 1 thread: Bad because you use only one core, but consistent performance assuming you're not on a multitenant 1X or 2X dyno.

- Clustered, 1 thread per process: Bad because requests are queued at the process level, so a slow request will cause any others sent to that process to have to wait. If you have a request which takes 25s to execute, the one sitting behind it in the queue is highly likely to timeout, and this can cascade to the next request, etc.

- Clustered, multiple threads per process: Better, but still bad, because while multiple requests will be running at the same time in the process, they are competing for the GIL and can slow each other down so much you'll still get timeouts.

The best solution we found to all this was to run a buildpack with a nginx reverse-proxy to a Unicorn server. It's nice because each request is run in it's own process, and the requests are queued at the dyno-level by nginx until any of the processes is free. Combine that with Performance dynos, so you can run a small number of dynos with lots of workers each, where each dyno has consistent performance, and the queuing issues mostly go away.

As you may be able to tell, I've spent far more time debugging this behavior than I would have liked to!

jordanthoms··on Patching Rails Performance
Speed improvements are always nice, but the big improvement for us came from moving from 1x and 2x dynos to Performance dynos. CPU performance on the 1x and 2x dynos was highly inconsistent, and would suddenly become abysmal for periods of 5-10 minutes or more - and due to random routing, the backed up dyno would keep on getting requests until the requests start timing out.
jordanthoms··on The WSJ was accidentally too honest about Uber in print
Actually, I believe they are highly profitable in the more mature markets. Overall they're losing money, but that's because they are expanding at an incredibly rapid rate.

I guess a potential problem will be if the smaller markets they are expanding into now don't turn out to be as profitable as the early ones once they mature.

jordanthoms··on Amazon details its plan for how drones can fly safely over U.S. skies
It's a possibility if the drones are delivering food for Amazon Fresh... That would be so awesome!
jordanthoms··on Why Docker Is Not Yet Succeeding Widely in Production
That looks quite nice in the few minutes I spent looking at it. The lack of Ruby support is a issue though, and we're also all on OSX and prefer CLI interfaces for most stuff.

Also, one of the killer features of Heroku (which few services seem to replicate) is log drains - I can easily add a http or syslog endpoint and have Heroku send the logs over. The other killer feature which isn't often replicated is One-off dynos, where we can spin up a new instance and get a console attached to it in one command - useful for running database migrations or using Ruby as a CLI to access data.

If we were on .NET that would probably be attractive, but it's still not really competing with Heroku.

jordanthoms··on Why Docker Is Not Yet Succeeding Widely in Production
I do agree with you, but we're on Heroku and that also has it's downsides - the biggest one we've found is that CPU performance on 1x and 2x dynos is very unpredictable - We've had to move to Performance dynos ($500/month each!) to get decent performance and prevent random timeouts.

Services like Cloud66 are interesting (they manage deployments onto your own EC2 or other cloud infrastructure), but the developer experience doesn't quite match Heroku yet.

Heroku really needs some more competition...

jordanthoms··on Amazon Aurora Now Available
Aurora has a bunch of nice features WRT automatic failover, redundancy, and automatically scaling storage as you push more data into the DB.
jordanthoms··on Why Britain has ‘ghost trains’
And not a totally unreasonable approach if the politicians want to maintain a specific capability, much better than making the entire market inefficient.
jordanthoms··on Ask HN: What are your biggest frustrations with Linux development?
Unfortunately, one look at that will tell you that it's no match for the rMBP in design or portability.
jordanthoms··on Coin Card Teardown
That'll change very shortly, first of my US credit cards was just reissued with a chip and you can expect that all of yours will be within the next year or so.

Note that it is Chip and Signature though, not chip and pin.

jordanthoms··on Outlier Detection at Netflix
This is very interesting and a source of frustration I have with New Relic and the other alerting services we use. New Relic uses a 5-minute rolling average for error rates, and alerts when that average goes above some threshold. However, that means that it takes ~5 minutes from a spike occurring to an alert being created - even if the error rate has increased to 50%.

It would be much better for it to be doing this sort of outlier detection - a gradual increase in error rate to 3% should not trigger a critical alert, whereas a big jump in error rates should trigger an alert quickly.

Has anyone implemented a system like this?

jordanthoms··on AWS CodePipeline
As far as I can tell this is more about orchestrating builds, to actually build your game you'd need to run jenkins on an EC2 instance and connect it to CodePipeline.
jordanthoms··on Did you buy that latte 2 hours ago? Think about leaving the coffee shop
It is, apparently.
jordanthoms··on Time Warner Cable owes $229K to woman it would not stop calling
Android 5+ (or maybe 5.1) includes this - you can set it to priority notifications only, and then turn calls on but say to only include calls from your contacts.
jordanthoms··on CVS Health Quits U.S. Chamber Over Stance on Smoking
Which raises the question of why so many of these people are vehemently opposed to e-cigarettes, which are essentially just a mechanism for delivering nicotine with some inert vapor.
jordanthoms··on New Zealand is making cyber bullying a specific crime
It's highly disingenuous to say that without also noting that this bill passed 116 to 5, and the most vocal critic came from the right side of the house. I oppose the bill as well, but we shouldn't pretend that National is solely at fault for it.
jordanthoms··on New Zealand is making cyber bullying a specific crime
One of the few Members of Parliament opposed to this speaking against the bill: https://www.youtube.com/watch?v=-5XZ7OZCzBY

"We must do something, this bill is indeed something, therefore we must pass this bill"

There are already sufficient protections in common law against harmful speech and passing vague law that only applies to online speech just makes us look like idiots.

jordanthoms··on Elasticsearch: The Definitive Guide
I actually have quite a few gripes with Elasticsearch. It's a pain to manage - there seems to be no way of killing long-running queries other than restarting it (so if you have users using kibana etc at some point someone will submit a impossibly hard query and kill the server), the quality of the search results just isn't that good and it takes a lot of customization to improve it - I've never been impressed with the results from any of the large deployments (e.g. github). Ideally it would come with good tokenization and synonyms etc out of the box rather than every deployment having to tinker with settings until it seems to work.

That said ELK is great overall and I would still use it for user-facing search, but I think there is still a lot to improve.

jordanthoms··on Safari is the new IE
The big difference is that both Chrome and Firefox update very regularly, and a high percentage of users keep at least reasonably up to date - the biggest issue was really that IE had massive market share and was also not updated.

We use a lot of chrome-specific APIs, and they allow us to achieve things that otherwise we couldn't (e.g OCR running in the browser through Native Client with no performance penalty vs a native binary). We're often implementing improvements with APIs that are a few weeks or months old. We do test on other browsers though and make sure things either degrade cleanly or we provide an alternative implementation.

jordanthoms··on Safari is the new IE
I think it's just a lack of optimization for battery usage on OSX Chrome. Google seems to have recently realized it is an issue though and is making progress on improving things: https://plus.google.com/+PeterKasting/posts/GpL63A1K2TF
jordanthoms··on Uber Bonds Term Sheet Reveals $470M in Operating Losses
This is a often understated advantage which Uber brings - the ability to land anywhere in the world and, modulo internet connection, get a safe ride within minutes where you know you won't be overcharged or longhauled.
jordanthoms··on Uber Bonds Term Sheet Reveals $470M in Operating Losses
In NZ the advantage is convenience and cost - you can get a ride far faster, payment is automatic, and the rates are lower, especially for shorter trips, than the incumbent taxi companies.

Some of the local competitors have some of that (e.g. Zoomy), but they charge extra fees for paying though the app which makes them expensive, and also means you need to go through more of a process when the ride is finished.

jordanthoms··on Shoplocket is shutting down
Many of the startups selling to megacorps where the product dies would have been dead in short order anyway, and I wouldn't assume everyone is getting a big payday.

It doesn't seem totally unreasonable to me for failing startups to sell to a larger company and (hopefully) extract some of the value left in it for shareholders and employees - I just wish that with that the acquiring companies would do more to support the customers of the startup with reasonable shutdown notices etc. Unfortunately nobody involved has a very strong incentive to do so.

jordanthoms··on Shoplocket is shutting down
Still, a month and a half is a very short transition time for a ecommerce product. It's disappointing when companies throw their customers under the bus, it risks making things harder for everyone.
jordanthoms··on Google Play Music – free, ad-supported radio
I.E the 'Google Photos' treatment. Now that Google's got the playbook for that rebrand down, hopefully they'll apply it to some more of their overlooked products.
jordanthoms··on SSD Prices in a Free Fall
Yeah, but the problem is that SSD failures usually have nothing to do with flash wearing out, and more to do with catastrophic, sudden failures somewhere else (mostly the controller AFAIK) causing the drive to stop doing anything at all. I had a 840 Pro fail on me, hadn't had many writes but just stopped showing up in the BIOS one day.
jordanthoms··on Why can't America have great trains?
America _does_ have great trains. The world's best, in fact. They're just used to carry freight, not passengers, which actually makes much more sense for most of the routes given the distances involved. [1] I can't find a more recent direct comparision, but in 2000 38% of freight in the US went by Rail, as opposed to 8% in Europe - by 2010, that rose to 43%. At PPP, US Rail transport is the cheapest in the world, including China.

There are areas of the US where high-speed passenger rail would make sense, and it's a shame that more hasn't been done in that area, but I do think that since the industry was deregulated the market has done a good job of concentrating rail investment on areas where it actually makes sense.

1 - http://www.economist.com/node/16636101

jordanthoms··on Stripe Is the New PayPal
When you are vague and refuse to elaborate, it can only lead us to think that your business isn't as squeaky clean as you make out.
jordanthoms··on “Fashion Is Hard. PostgreSQL Is Easy” [video]
Postgres replication is probably not as easy to setup, but can be extremely powerful especially when use tools like WAL-E as well.

We have a setup where our WAL log is rotated every minute and backed up to S3. Replica databases can either load from S3 or connect to the master for streaming - so they will catch up using S3 and then connect, reducing load on the master.

jordanthoms··on Introducing Empire: A Self-Hosted PaaS Built on Docker and Amazon ECS
Interesting! We are in a similar situation to where you were, with a bigish app on Heroku which we are keen to move over to EC2 to join the rest of our infrastructure, definitely keen to see how empire develops
← PreviousPage 10 of 27Next →