My understanding is that you're describing the CIE 1931 color space:

https://en.wikipedia.org/wiki/CIE_1931_color_space#Color_vis...

You'll see it is based on the physics and human biology, and is the basis for everything else.

The thing is, this color space isn't very useful for color calculations in the perceptual/subjective sense (e.g. if I just want to change one characteristic, like luminosity, without affecting the chromaticity), so we have transformations to more useful spaces like XYZ, Lab, etc.

There's also the fact human vision is a subjective/psychological phenomenon, so only frequency response is not enough: our vision can map different frequency responses to the same perceptual color (metamerism), our vision adapts color perception based on light source, etc.