There are much more interesting ways to relate numbers to pitches; see the Pythagorean ratios and tuning before equal temperament.

http://en.wikipedia.org/wiki/Pythagorean_tuning

When 1-2-3-4 is used in music to show degree, that is an ordinal representation, not a cardinal representation. That's sort of the key difference here.