Are you suggesting to style the tag name directly? Because that would create chaos in a codebase. <button> is for semantics. It's possible that other design system components need the same semantic element.
And as you say, it’s for semantics so I wouldn’t use .button anyway as it doesn’t carry the right specificity for styling.