There are randomised controlled trials of diets (without caloric restriction). They show that some diets are better than others for real clinical endpoints, eg mediterranean vs low fat diet for secondary prevention of cardiac events [1]. The effect size in that example is not small, it is large, despite most participants already being on a statin. This data shows that diets are biologically active and significantly impact health. Therefore, the philosophy you espouse cannot be correct, or at least needs to be heavily modified.
[1] https://www.thelancet.com/journals/lancet/article/PIIS0140-6...