It's not just about what we are used to. The argument about where the "central" point in the scale matters (for seconds and for grams) because the prefixes are by their very nature logarithmic, with "easier" distinctions made close to the center point/0-point on the scale axis than further away in the scale.
milli-, centi-, deci-, deca-, hecto-, kilo- are all six very close surrounding the 0-point/origin. Everything else gets logarithmically further out.
Using second as the 0-point is great for computation, as milliseconds and even microseconds can matter to a computer, but it isn't great for the human perception of time where people are bad at even granulizing events that happen in deciseconds, leaving nearly all of the decimal prefixes almost useless in human timescales.
Which is why you see the French Decimal Time system mentioned in the article here and used by the poster above in this thread centered around the day as its 0-point. milliday, centiday, deciday, decaday, hectoday, and even kiloday are all relatively useful units in a human timescale.
While you are correct that we don't "need" that kind of vocabulary, the argument is that it would be useful in conversation and in day-to-day (fig. and lit.) math. Having additional prefixes serves as mental anchors/frames for the conversation. If I give a number of seconds to you as hundreds of milliseconds versus deciseconds, that gives you additional context about my methodology and/or focus in whatever that benchmark was. Sure, you can easily convert between the two prefixes, but that context can be useful.
Just as the difference between if I tell you something is happening in hundreds of minutes from now versus threes of hours from now. There's probably a useful contextual distinction there. That's why people play with solutions like Decimal Time changing the scale altogether away from seconds to something more like an Earth day, or in the other direction explore ideas with more prefixes to fill in some of the logarithmic "gaps" such as myria- and hebdo-.