The fact that people find it so confusing is one. If people intuitively think that "width" is the total width of an element (an entirely reasonable assumption, mind), then that would imply that CSS chose an unnecessarily confusing name.
Another is from the mental load perspective. If I'm doing a layout, I want to figure out how things fit by looking at the total space they take up. _That_ is the width I care about. Leaving margin out of it makes sense, but having to mentally add the padding and border to get from "width" to "no really width" is annoying.