While your point is literally true, having basic arithmetic proficiency does go some way in making sense of the world. Quick mental calculations to see when you'll reach a certain place, ability to compare things when they're in different units etc. It's not only the skills but the overall mental model that they help create. The world starts to become a little more rational and sensible when you can do this.