Admittedly, when self-hosted some of these may go away, but here are a few that come to mind.
* General awfulness when it comes to packages. Outdated packages. Nonstandard packages. Ubuntu, instead of debian + testing, is a terrible choice for a CI.
* Horrible support for multilingual builds. God forbid if you have Python 3 scripts in your go project.
* With github, force-pushing during a build causes the build to error and travis can't detect that
* Lots of papercut issues in the UI
* There's always something when setting it up on a project that makes me end up fighting with it and figuring out crazy workarounds for it for several hours.
There's a lot more, though most of them have to do with point #1 and their choice of Ubuntu as a platform. I don't remember the rest.