Hacker News Reader

Comment by gkoberger | Hacker News Reader

I'm historically a huge Heroku defender. Been using it for a decade now, and my company still uses it despite being quite large and getting a lot of traffic. It's always been a great product and the early Salesforce days they DID ship a ton of new stuff and improve rapidly (despite a narrative that they didn't). Like, it got REALLY GOOD the years after acquisition.

That being said... it's insane that we haven't been able to deploy for over two weeks and nobody there seems to care. So we're looking to move now, since it's clear Heroku has pretty much just given up at this point.

View on HN

If anything, the number of outages in the past two weeks on https://status.heroku.com/ has increased to the point where colleagues have asked if they can mute our #devops-alerts channel subscribed to notifications there. That's the canary in the coal mine if there ever was one. I have to imagine that rushed updates and maintenance windows are in reaction to the massive leak of credentials a couple weeks ago - but in a well-run organization that would trigger a need for additional caution, not less. Something is horribly wrong at Heroku right now.

I'm in the same boat. We were in the process of moving off of Heroku a few months back. We had to pause that migration for a number of reasons, but the initial jump was to save some money.

Now I'm kicking myself for not pushing the migration to completion. I've basically had to spend the last week recreating much of our deployment pipeline using a very complicated local deployment structure that only I can execute. It's a complete nightmare. My only guess is that the Heroku team is just a handful of overworked developers. For the Github integration to be down this long they must just not care. Like at all.

I feel your pain !

Never underestimate the value of a "good enough" bash scrip to deploy ALL your project(s).

First (ok top 5) thing i do when starting any new project is quickly writing a "bash deploy script" which usually goes like this:

  *build-for-prod

  *post-build-steps (zips, uglify,spit-and-polish etc)

  *tar everything

  *scp tar-file to server(s), extract, restart xyz

Sure it's unsexy as hell in today's world of <insert-fav-deploy-tool-here-that-calls-bash-things-underneath-anyway> but it works so nicely !

Interesting side story, about 10 years ago I was employed by a "big"(for me at least) price comparison service, and for 5 of the 7 years we use a simple bash script to deploy most of our API's and frontend.

Everyone agreed (it's the wrong way) to do it but no one wanted to dive into the alternatives.

We even found a bug where there were some race issues (some weird service configs) when we deployed that it only worked every 2nd time ? horror

So we just "always deployed twice"... since it's was super fast !

I did this sort of stuff for a time, it's even better if you link webhooks to your git provider and make those fire the actions directly in the server.

Once the script was set up to do all the build and deployment, we only needed to push to the release branch to load it, just as painless and with more or less the same underlying ideas that the easy CI/CD you get with providers like Heroku et al.

It still really pays off to know a thing or two about sysadmin nowadays, literally and figuratively.

> we haven't been able to deploy for over two weeks

What is stopping you? Are you unable to type `git push heroku master`?

That's fine as a solo developer – but what about a team with CI pipelines etc?

can't you put that in the ci pipeline

I’m pretty confident if ‘putting it in the ci pipeline’ were a straight-forward option for most people, they probably wouldn’t be paying Heroku to manage review apps. I’ve used Heroku review apps for years and have also written and taken over different custom deployment pipelines. Review apps have a million different ways to be a giant time and money sink if not planned and implemented properly.

I think you're underestimating the complexity of our codebase. Aside from the usual blockers, we have hundreds of repos (our Staging feature for Enterprise allows customers to deploy our code on their own cadence) all tied together using pipelines.

We have our own custom release management software, which now doesn't work. Different repos have to go out at the same time so things don't break. Plus, we extensively use their review apps for code reviews, which we've lost access to.

Lastly, not everyone has access to deploy directly to Heroku, so not everyone would be able to 'git push heroku main'.

Could we fix all of this and get it working? Yeah. But we want to be focusing on building our product, which is why we pay Heroku a ton of money so we don't have to worry about this.

We were in the same position (although luckily far fewer repos than you!). It took a bit of fiddling, but in the end I found that it was actually quite easy to fix this by tacking on a force push to the main branch of the heroku git repo at the end of our existing CI process (essentially treating heroku got as a deploy api that happens to use the git protocol)

Don’t blame you for wanting to move, but you might find that approach helpful as a quick fix.

Sounds like you needed to start look into migrating into a hosted k8s solution (AWS or whatever), which will probably be quicker than waiting for Heroku

Edit: ah I see you're looking into moving already

This doesn't work if your Git repo is above a certain size. Some of our apps (fortunately not production) haven't been able to deploy since the incident.

* It was automated. Now it requires someone to pay attention and do it. You need to check if CI passed, and pull, and be sure you pulled the version that passed CI (maybe someone pushed since then).

* Review apps, the only remaining heroku "killer feature", do not work at all.

The fact that this has been broken for 2 weeks tells you everything thing you need to know about the state of their code base, and the resources salesforce is willing to allocate to heroku.

Has anyone switched to AWS App Runner? Curious how it went.

Tip for your first point: make the CI server push to heroku.

^^ This.

ex-Heroku here.

The codebase is sound. I would almost certainly expect that the reason for the slowness is the ability/diligence and paranoia levels the SFDC security teams have. They won't want to turn this back on until they are absolutely certain it's 100% again.

Heroku's github integration has been down for 2 weeks

That doesn’t concern me as much as how our github access keys got accessed/leaked which Heroku has yet to answer.

Reply on news.ycombinator.com