I fully concur with your last statement. I find that the main advantage of having less moving parts is that there's exponentially less things that can go wrong - and that makes reliability possible with very small teams. The setup cost is usually not steep when compared to the overall lifecycle of a project, I fully agree.

The other argument I have against many DevOps tools (and most software tools in general) is that the patterns of the solutions they provide don't generally match the basic patterns of the problems they are solving - the consequence is the sheer complexity of understanding the tool, especially when something goes wrong. In the past, this hasn't only costed me time, but a lot of frustration that I'd rather avoid - even at the price of having to build my own solutions.