tomlemon

I take it you don't believe in the SIA? Is this paradox irreducible? Is the world going to end soon?

This annotation will always be on the version of active_model.rb at master, even if it changes.

And we try pretty hard to anchor annotations even when the underlying text changes. Here's a talk I did explaining our approach to "fuzzy annotation anchoring" if you're super interested: https://www.youtube.com/watch?v=FJyqfRcyYIQ

But it's definitely not perfect! Hit me up with suggestions: tom@genius.com

For example: http://genius.com/5077469/gist.github.com/RogerPodacter/3e8a...

And actually you can put "genius.com/" in front of ANY url to annotate that page. For example: http://genius.com/5077492/www.paulgraham.com/yahoo.html

There's also a chrome extension: https://chrome.google.com/webstore/detail/genius-beta/ccaokn...

You can sign up for the beta at http://genius.com/beta.

(Questions / feedback welcome – tom@genius.com)

Likewise with "It's not not your job"; there's definitely a conflict between "take ownership over the overall success of the business" and "focus instead of trying to do everything all the time", and I tried to nod toward this, but I do think both sides are true! You do want to focus AND you don't want to be saying "Well, the company failed by at least I did my job".

Anyway, these are definitely far from perfect but I'm throwing them out there anyway because I think they're better than nothing.

I'd love to see your criticisms in the form of annotations on the piece itself tho! Everyone should get to read them!

The fact that all our text and song page URLs end in "-lyrics" is a relic of a time when we hosted almost exclusively lyrics.

Making the changes required to host all kinds of annotated texts has been a big project and making the URLs make sense is something we haven't gotten around to.

The reason it's not a totally trivial change is that both users and texts share the same top-level namespaces, and so right now we use the "-lyrics" suffix to differentiate song and text pages from user profile pages.

We want to change the suffix to "-annotated" for non-music pages and will hopefully get around to in the next few weeks. To our knowledge the existing situation doesn't help with SEO (no one is searching "bartleby the scrivener lyrics"), and it's confusing, so we 100% agree that the current situation is bad.

For more info on the "Worse is Better" philosophy that caused the current jank situation to persist for so long, see this talk: https://www.youtube.com/watch?v=X45YY97FmL4

We didn't know how bad the performance was because Heroku's tools (logs and New Relic) reported incorrect performance data (i.e., they said requests weren't queuing when they were actually spending a ton of time queuing)

EDIT:

> Would Rap Genius still have sued if the documentation had been 100% correct and instead the problem was just plain old slow I/O on Heroku's side?

To be clear, we have no official association with http://herokuclassaction.com/ or the lawyer behind it – I actually found out about the site from the article's author when he interviewed me for the story.

(But I do think Heroku owes its customers a refund)

Tom@Rapgenius| about 1 year ago I know this is a bit of a vague problem, but I've been getting a bunch of Error H12 (Request Timeout)s recently, and I'm not sure what to do about it. It's not like I have some particularly slow actions; I'm getting this error for actions that under most circumstances work totally fine (i.e., return in less than 300ms). Also I don't have a deep request queue (I'm running 40 dynos which is more than enough). Maybe I'm doing some slow queries? Should I upgrade my DB? Also, I do notice that most of my app's time (according to New Relic) is being spent in Ruby (http://cl.ly/29132F272W2D0K1l2I3P). Would upgrading Ruby to 1.9 noticeably help this performance? (I'm a bit nervous it'll create a ton of problems).

Phil@Heroku Hello - I can look into this, but I'll need access to your New Relic account. Will you make sure 'phil@heroku.com' has access? Also, from your screenshot I notice your DB times are ~ 100 ms. We recommend keeping those times closer to 50 ms. You might be able to speed things up with a database upgrade. I'll look into New Relic once I have access and let you know what I find.

Tom@Rapgenius Thanks, Phil! How do I give you access to my New Relic account? I tried clicking "account settings" and got this: http://cl.ly/0V2J3i0826400I2s3b2c

Phil@Heroku Tom, I have access now. I'm not sure what was blocking me earlier. After looking at New Relic and the database server, I think a larger database will help. At the very least, it will be helpful to try the next level for a week and compare performance statistics in New Relic with the prior week. Your app is using an Ika right now, and the next step up is the Zilla database. We've made the upgrade process very simple, and it's outlined here - http://devcenter.heroku.com/articles/fast-database-changeove... Your database is ~ 5.4 GB in size (via the 'heroku pg:info' command) so an upgrade shouldn't take too long. You will be able to test the process by adding a Follower and timing it via the 'heroku pg:wait' command. This should give you a good idea of how long it will take to spin up the new database. Also, should the Zilla not help much, the downgrade process to an Ika will be the same. You only pay for the resources used.

The current database server appears to be a bit under-powered when it comes to Compute Units. The Zilla has more power and should provide some room to grow. As for an upgrade to Ruby 1.9.2, I'm not sure how much that would help. It would be an involved upgrade that would take time to plan and deploy. The database upgrade should be a quicker solution. Long-term you may want to consider moving to the Cedar stack and Ruby 1.9.2.

Tom@Rapgenius Thanks! I'm upgrading now

Tom@Rapgenius I'm still getting a ton of "Request Timeout" errors. E.g.: 2011-12-08 14:46:53.222 219 1 2011-12-08T14:46:53+00:00 d. heroku router - - Error H12 (Request timeout) -> GET rapgenius.com/Wale-ambition-lyrics dyno=web.17 queue= wait= service=30000ms status=503 bytes=0 one weird thing: there aren't any values listed for the "queue" and "wait" parameters. Could that indicate a problem? Could an exception have been thrown earlier in the request before the timeout? Or does the timeout error just indicate that the request took too long? If it's the latter I'm not sure how to troubleshoot all these errors since the associated actions are fast the vast majority of the time

Tom@Rapgenius Here's another interesting example:

2011-12-08 15:59:32.293 222 1 2011-12-08T15:59:32+00:00 d. heroku router - - Error H12 (Request timeout) -> GET rapgenius.com/static/templates_for_js dyno=web.17 queue= wait= service=30000ms status=503 bytes=0 This action is extremely simple – it doesn't access the DB or any external services. Here's the template: Ballin! <% unless current_user %> <% form_for User.new, :html => { :id => '' } do |f| %> Tired of entering your email address? Create a Rap Genius account and you'll never have to worry about it (or anything else) ever again: <%= render :partial => "/users/form", :object => f %> <%= f.submit "Create Account" %> <small>(Already have an account? <%= link_to 'Sign in', login_path, :class => :facebox %>)</small> <% end %> <% end %>

Besides a big request queue (which there isn't), how could this action possibly time out?

Phil@Heroku Tom - sorry for not getting back to you sooner.

It's possible for H12s to occur even for simple actions if there is already queueing for the app. With a busy site like your's, even a few H12s can cause a cascade of H12s for successive requests.

It looks like New Relic has not reported any downtime over the past 24 hours. Can we let the site run through the weekend and see how things look Monday after 3 days of New Relic data with the new Zilla?

Tom@Rapgenius > It's possible for H12s to occur even for simple actions if there is already queueing for the app.

I feel what you're saying, but I don't think my app's queuing. For one thing, New Relic shows 0 time spent in the queue during the period in which I'm getting all these timeouts. For another, I'm running 40 dynos and my average request time is <400ms. So:

400 ms * 3000 requests / minute * 1 min / 60000 ms = 20 simultaneous requests (i.e., 20 dynos) so 40 dynos should definitely be more than enough.

Also, shouldn't Heroku be showing me the queue / wait stats at the time of the timeout? That would help prove whether my app was queuing at the time in question

It looks like New Relic has not reported any downtime over the past 24 hours.

New Relic isn't great at catching intermittent problems like this; you really feel it when you're using the site continuously for an hour or whatever. Also, users make many more HTTP requests than New Relic (since every page load kicks off several AJAX requests).

That said, there has been downtime in the past 24 hours (though less than in the previous 24): http://cl.ly/3D2b1Z170B0w1f1m113m

Tom@Rapgenius Here's some additional data: At 5am this morning (EST), Rap Genius went down. I woke up at 11am (it's a Saturday!), did a logs --tail and observed that basically every request was timing out. I did heroku restart, and now every request started returning a backlog too deep error

Finally, I added another 10 dynos (bumping the total to 50, which is a log of dynos!), and this seems to have fixed the problem – perhaps because my app needs the additional capacity, or perhaps because merely changing the number of dynos reset something else. Either way, I'm sticking with 50 dynos for now out of fear even though I doubt my app needs that many (right?)

Either way, the 5 hours of unexplained downtime (there weren't any application-level exceptions or anything) that was fixable by tweaking my dyno count further supports my theory that something's going on with my app on Heroku's end.

Phil@Rapgenius Tom - I've been looking over your New Relic stats.

First - the good news - the upgrade to a Zilla seems to have helped. Database times are down a bit, which can only help. I checked the actual database server and it's not showing signs of over-work like the previous Ika was. Second, I notice that downtimes reported by New Relic over the past two weeks are in the early morning hours - 3 to 6 AM PST. Do you have any scheduled tasks that run during these times?

Also, request queueing is nearly zero, so 50 dynos does seem like a lot. What are your usage patterns like? The RPM graph in New Relic indicates the normal cyclical usage pattern, lower during the night, but what does Google Analytics tell you?

Finally, the Heroku platform has been having issues over the past week, but none of them correspond to the downtime you had Saturday morning.

Just put it up on Rap Genius – create an account to help explain this post to the Heroku users it affects!

If Heroku actually knew about the problem for a long time and yet didn't officially respond or apologize until we published that post, doesn't that make you take their apology less seriously?

> Especially the one line about the date of the report, that's just pedantic.

What they claim to have known and when is very relevant to how much you can trust them! Here's the timeline from my perspective:

1) In 2011 Tim Watson points out the problem: http://tiwatson.com/blog/2011-2-17-heroku-no-longer-using-a-...

2) Heroku responds but does nothing (https://groups.google.com/forum/?fromgroups=#!msg/heroku/8eO...)

3) On 2/8/2013 I send Heroku full details about the problem including simulations, etc

4) On 2/11/2013 Heroku responds and says the best path forward is for me to optimize Rap Genius and that Adam's done talking about it with me.

5) The Heroku's Ugly Secret article goes up on 2/13/2013

6) Heroku releases BIG apology on 2/14/2013 saying they only learned about the problem on day ago.

Heroku knew about the problem at the LATEST on 2/11, and before we made a big public stink, their response was to say "just make Rap Genius faster"

1) Understates the problem – It affects not only Bamboo, but all thin (and other non-concurrent web servers) on Cedar. And since thin is the default on Cedar, the problem affects all apps on Cedar by default

2) Understates how long they knew about it – I notified Heroku about the problem INCLUDING the simulation results 3 days before the blog post came out and yet the apology claims they didn't know about the problem until seeing the blog post.

There are definitely ways to improve the performance, but you can't measure them since Heroku gives you no way of determining how much time requests spend in the in-Dyno queue. Tho you can modify your app to get New Relic to display this info: http://rapgenius.com/1506509

> What seems particularly odd is that rap genius appears to be using their Heroku dynos to serve their css, javascript and certain images. So loading the homepage appears to make about 12 requests that hit their dynos, rather than 1.

These assets are Cached in Varnish. We serve them without hitting dynos at all

Our response: http://rapgenius.com/Oren-teich-bamboo-routing-performance-l...

> The simulation was a bit of a stretch because the supposed number of servers you need to achieve "equivalent" performance is highly dependent on how slow your worst case performance is, and if your worst case isn't that bad the numbers look a lot better

It's still pretty bad. Here's a graph of the relative performances of the different routing strategies when your response times are much better (50%: 50ms, 99%: 537ms, 99.9%: 898ms)

http://s3.amazonaws.com/rapgenius/1360871196_routerstyles_fa...

See http://rapgenius.com/1504222 for more

Random routing is still wayyyyy worse than intelligent routing even if you're processing multiple requests simultaneously on a single dyno (see http://rapgenius.com/1504222)

Requests will still queue, but since time spent queuing at the dyno level doesn't appear in your logs or on New Relic, you'll never know it.