It's hard to know exactly goes into the credit score calculation since the algorithm is proprietary but, anecdotally, you can carry a small balance (e.g. <10% of your credit line) without it having a negative impact on your credit score.
For example, if I have a balance of $800 when my statement was generated, my bank is going to report that $800 to the credit bureaus and FICO will calculate my credit score accordingly. So, for this cycle at least, it doesn't matter that I pay in full before my due date (thus avoiding any interest expense), my credit report will show an $800 balance on my account. This is why it's more likely to experience big month-to-month jumps in your credit score when you have fewer accounts/lower credit lines.
I do agree with you, I think that not carrying a balance doesn't have any negative impact on your credit score.