For us it was mainly the fact that every month some part of Heroku is down or degraded. That and the huge price increase when jumping from the standard dynos to the large dynos.

For reference, we migrated to new t4g instances on AWS and received about a 100x performance improvement for a similar (sometimes cheaper) price than Heroku. We are also able to connect to our company VPC and use private resources partitioned elsewhere in AWS (e.g. a redis cluster that costs the same as Heroku with far fewer limitations).

The downside is obviously configuration and learning AWS. Thankfully, we are pretty well versed in AWS here.