Elevated error rates on applications and deployments | Hacker News Reader

Elevated error rates on applications and deployments

status.heroku.com

93 pointsjorrizza69 comments

I'm currently at the Nordic.js conference http://nordicjs.com/ (which... is down right now). Suddenly half the audience started to looking at mobile phones, unpacking laptops.

Poor timing for the current speaker...

At least our apps are now up and running again after being down.

Finally I can start focusing on the talks instead...

The Erlang User conference is next door, there'll be some people leaving as well

Down for almost an hour now. Haven't seen anything of this scale happen in 18 months or so of running our apps on heroku. Looking forward to the postmortem from Heroku (and hopefully my app being up again).

Do they really release postmortems? I've seen them mentioning postmortems in incident statuses, but I've never actually read one.

https://blog.heroku.com/tuesday_postmortem

October 27, 2010. Really? :)

Heroku totally releases postmortems for incidents that meet certain criteria but I can't speculate as to what this criteria might be today.

Here is a recent example: https://status.heroku.com/incidents/930

Source: I used to work at Heroku on the team which managed this process.

It's more than some apps... all of ours (~70) are down and deliveroo looks like it's also having issues

Yep, people are melting down on Twitter about not being able to order food. And on the day that Amazon launched Amazon Restaurants in London with one hour delivery too...

What options would you consider if heroku was down permanently ?

If you are just using heroku to run a process it is pretty easy to rebuild on AWS. Here is one fairly solid guide: http://blog.clearbit.com/ec2-heroku/

Depending on what language you are using google's app engine could be a quick replacement.

http://rogerstringer.com/2015/05/13/make-your-own-heroku/

https://github.com/buttfoundry-community/bosh-buttfoundry/bl...

Another HN thread: https://news.ycombinator.com/item?id=4183634

I'm not a big fan of docker for a few irrelevant reasons but using it w/ heroku makes it very easy to migrate to other container hosts.

edit: the founder of Flynn replied to parent as well, that is also a great alternative that I forgot to include.

Please do not trivialize the aspects of running a server on AWS (or a highly-available cluster) just for your own single application. Installing is the easy part, the day to day maintenance, reacting to issues, the 3am alerts (you set up alerting, right?) is what's costly.

I guess I was responding to the parent with ideas and links to use 'in case of emergency'. It isn't trivial, but some of these tools in conjunction with AWS make it possible to do alone, which I think is pretty cool. Alerting? Cloudwatch + PagerDuty gets me somewhat there.

(Disclosure: I'm the CEO co-founder of Flynn)

Heroku is a great product and it going down permanently would be a huge loss to the community.

However there are a number of open source alternatives with very similar user experiences that you can run yourself in a variety of environments including:

https://flynn.io/ https://github.com/dokku/dokku http://deis.io/ https://www.openshift.com/

From what kind of scale would you consider running flynn (I'm a fan since the early days) or deis (scoped it as well) a pragmatic step compared to heroku?

we (mindmup) moved from Heroku to s3+cloudfront for web pages and assets and AWS lambda for backend functionality, works like a charm.

It took them 15 minutes to update their status from the time of our last received request.

Just like with previous incidents. I understand operating Heroku is not trivial but with 15 minutes after my monitoring systems (Pingdom, New Relic and Logentries) notifies me they need to improve this.

also not the cheapest service and no direct support line…

What are you comparing to cost wise? Please do include costs for setting up and maintenance in that comparison.

As for support I have always found Heroku to respond very quickly. And they do have direct support. https://www.heroku.com/support

Agree, support is usually very fast. Also, an outage like today is quite rare, I'd say.

Perhaps it’s suitable for the US but in EU we are seeing this for the most of the working day:

"Heroku Support is currently closed. Normal support hours are 9 AM to 9 PM Eastern time. Our goal is to respond to your inquiry within 1 business day. Please note that it may take longer depending on current volume. Running a critical app?

Heroku's Premium Support offering includes a 24×7 Response SLA and more. Please contact us for further details."

When it is Heroku’s fault (couldn’t access my account with cryptic error message) it is very annoying to be suggested to use paid support, for a product that is already costs more than alternatives.

If it's a Heroku error then they obviously have staff monitoring servers 24x7. Usually I get very quick responses from Heroku Support at all times during the day, regardless of the type of inquiry - either basic questions or platform problems. Obviously they would filter incoming requests. If you pay for support you get a quick er response. If you have a question without paying you might have to wait a bit longer during off-peak (US) hours. But if it's a platform error I'm pretty sure they would filter that out and respond quickly.

And what are you comparing to here? Are you getting instant 24x7 free support from any other vendors? It's quite normal to pay for support in this industry.

What is a good alternative for static single-page apps? What are downsides of S3?

I host all my APIs on Heroku and all my static front ends on S3 with CloudFront on top. No downsides in my mind. Free and instant TLS certs with CloudFront, too.

And you can even get a free domain name through http://www.freenom.com/

Disclaimer: not affiliated

Many options. S3 is probably the most solid. Consider GitHub/GitLab pages. GitHub is pretty solid. GitLab allows you to use things other than Jekyll.

My search led me to Netlify, which seems pretty great. It runs your build script when you push to GitHub, then serves the resulting static files with a Let's Encrypt certificate. Comes built-in with a CDN, so you don't need CloudFront or CloudFlare.

I've recently been converted to Netlify after fighting hard against moving off of straight S3. There's an awful lot to love there.

Take a look at netlify (https://www.netlify.com). We're like a heroku tailored for that stack.

I use Github pages for my personal site.

Aerobatic is one alternative option. Git push deploy. NPM auto-builds, CDN, SSL and more. https://www.aerobatic.com/

Disclaimer: co-founder at Aerobatic

Uptime robot is reporting my sites went down at 12:55 UTC, so been down for 35mins

A follow-up report has been included on the incident page: https://status.heroku.com/incidents/941

It's been down for more than half an hour now... don't understand how it took them so long to update their status page.

I'm not sure why, but I was under the impression that Heroku had been shut down months ago.

I may have been thinking of Nodejitsu (which has been acquired by GoDaddy). Did Heroku change owners at some point or am I imagining things?

Heroku was acquired by Salesforce in 2010.

Heroku was acquired by SalesForce, and like everything Salesforce touches, it'll slowly descend into a steaming pile of

So far it seems that Salesforce is leaving them alone to run their own business

They're one of the only acquisitions that have been kept running pretty much the same way.

There was no integration with force.com or anything like that. I don't think they're even colocated in SFDC datacenters, I think they still run on AWS

Heroku has been owned by Salesforce since 2010, and it's very loved in the SFDC family.

No chance of it shutting down anytime soon.

Heroku was bought by Salesforce in 2010: https://techcrunch.com/2010/12/08/breaking-salesforce-buys-h...

I got my first SMS alert notification from Pingdom about 55 minutes ago now. One or two requests managed to reach through the the application logs in the last 10 minutes, but still mostly down.

Seems to be resolved now...

Seems to be coming back up now, albeit a bit intermittent.

And we're back.

ish.

Deliveroo's error message is on point.

https://deliveroo.co.uk/

What was it?

Nginx default error page, this one: http://i.imgur.com/IS9toHS.png

Back Online!

Up now

This is kind of offtopic but what is the attraction of using Heroku over using EC2 directly now. I remember back in the day when EC2 didn't have RDS and Elasticbeanstalk Heroku was an attractive option because you could deploy and scale without needing to do any kind of system administration.

But now EC2 is offering managed databases through RDS and elastic bean stalk gives you git style deployment similar to Heroku I don't see what Heroku is offering other than another point of failure and another set of security problems. It looks like Heroku uses linux containers for isolation. So not only do you have to worry about someone attacking the underlying EC2 VMs Heroku uses but you have to worry about the tenants collocated on your Heroku VM attacking you through the linux kernel as well.

Elasticbeanstalk is a huge mess when you look at its guts. Deployments aren't any better than Heroku either.

Speaking from the perspective of someone getting small projects off the ground, the value of Heroku (and similar) is that they help you stay focused on dev rather than ops.

In your example of security, you take an increased risk due to attacks on shared services, but I'd suggest that only a small fraction of "production" apps can field as qualified security teams as Heroku can provide... so if Heroku's customers all moved to EC2 there'd be an increase in badly managed and unpatched instances.

It's still significantly easier to get up and running and keep a system up and running on Heroku than the AWS equivalents.

I'm using Heroku right now and have thought about switching to EC2 as it would save me a bit of money. But the reason I haven't is simply time. Heroku's Pipelines feature, integrated with GitHub, means I get review apps and automatic deploys on merge, just by clicking a few buttons. Now I know this is completely doable with AWS but the time to set it up and switch would be more than it's worth right now.

I don't see Heroku as a very long term solution for us but for the time being, while still a one-man tech team, it just means one less thing I need to deal with. Until I do ...

I wrestled with Elastic Beanstalk 10 months ago for 2 days and had to give up in the end. Every deploy was dog-slow. And I was trying all kinds of things, imagine the frustration. Only certain versions of software are supported, though I downgraded just to test EB. But I still couldn't get it running in the end.

I can't recall a single time I had a problem deploying to Heroku, including the first deploys. Heroku is lightyears ahead of EB. Speaking of deployment, I might try Convox sometime in the future.

OTOH you can use RDS with Heroku, just be careful to have them in the same region.

EB supports docker deploys now, which helps with the version stuff, though can create it's own difficulties. True that EB deployment is very one-function though. Tacking anything on is hacky and gets thrown behind the scenes, so is hard to track and audit

There is much more to Heroku than managed database and git-based deployments. For instance, recently they have seriously improved the CD experience with Pipelines, Review apps and GitHub sync. They also have pretty seamless integrations with hundreds of providers (email, logging, search, monitoring, ...) in the form of add-ons. You get some basic metrics on your application.

And if security is a concern, or/and you want your dynos to talk to each other, they now have Private Spaces (https://www.heroku.com/private-spaces).

Simply said they are taking the "managed services" experience to a whole new level imo.

Plus, you may not care about that, but they actually have a UI/UX team that can design beautiful AND usable dashboards.

(Note: I don't work at Heroku, just a happy user)

This exactly. It essentially takes care of all the setup for me, plus the CD and provider add-ons are incredibly easy to install and setup.

I basically outsourced my cloud platform developers.

True, and that's why I stuck to heroku for a long time.

But, in the end I realised I can replicate most - if not all - of it with docker cloud and github. Most CI services support github anyways, so that's also covered.

And it's significantly cheaper, for my case we have 90% price reduction - paying only for instances + docker cloud.

Not connected with any service.

How does Heroku's CD experience compare with AWS CodePipeline?

Simple. For me, I have a dozen apps running on heroku, and they are all on the free tier. The amazon free instances require credit card and only last for a period of time. I've had mt heroku apps running for years.

It makes spinning up and trying an idea very fast.

For the projects that end up having lots of users, I move to GCE or AWS

I use it when I don't want to spend time with servers or if the thing I'm deploying I don't trust - can be easily exploited. Like recently deployed a phantomjs service and well I'd rather not be concerned about the security implications of running a web browser on a server

For sites already using Salesforce, Heroku Connect is quite a time saver. It handles the two-way sync between Salesforce objects and Heroku Postgres for you.

It's not a time saver, unless you are small and one developer works on both sides of integration. Integration through database is an antipattern, because it makes apps strongly coupled and fragile and building an app on top of Heroku Connect database adds extra management and maintenance costs, required to keep both sides in sync.

It's definitely an antipattern, but if you're unfortunate enough to be on Salesforce it certainly beats writing in Apex (which also exposes you to the underlying object schema anyway).

I wrote up a pretty long blog post for Codeship that generally explains it: https://blog.codeship.com/exploring-microservices-architectu...

Reply on news.ycombinator.com