It is important to maintain the spatial structure of the units.
Energy and torque are both N.m in SI units, but in one case the vectors are aligned, and in the other they are perpendicular.
Energy is a scalar, torque is an axial vector.
Energy and torque are both N.m in SI units, but in one case the vectors are aligned, and in the other they are perpendicular.
Energy is a scalar, torque is an axial vector.
In some cases, like vectors and axial vectors, it is less obvious because (in 3 spatial dimensions), they actually have the same number of components. But still, if you add them together you get something that does not transform in a sensible way under reflections.
I suppose you could even go through the relevant laws of physics and only identify the units for quantities that need to be added together (or to make the quantities inside more complex functions unit-less).
In the case of torque and energy you basically run into the problem that torque times angle is an amount of work and angle is traditionally unit-less. You could however give angle a unit by introducing some 'unnecessary' conversion factor like 'degrees per radian', which is only used for trigonometry. In that case you could reformulate torque as 'energy per angle'.
Possibly of interest, as mentioned in Appendix C of https://www.nist.gov/publications/architecture-software-assi... there is a start to this in https://en.wikipedia.org/wiki/ISO/IEC_80000
The Gibbs formulation of 3D vector analysis/calculus was certainly useful, but forgets most of the spatial structure. Using Clifford Algebra (Geometric Algebra) fixes everything.
Almost all of the value in GA comes from the use of multivectors and the wedge product. It's very hard to find a justification for the "geometric product", and there's no real geometric interpretation of it. Therefore treating it as fundamental is a difficult-to-justify choice. "Exterior Algebra" on its own has everything you want without the objectionable parts.