Using your example, in the limit you end up with a data type carrying a chain of conversions with it, that reference some database that you don't own, but have to maintain access to. That altitude's reference itself is a particular geographical coordinate system (of many possible GCSes) that references a particular datum ("on what planet are we, and how does it look", of many possible datums), both of which are maintained by third-party organizations and political in nature. And then when you have another altitude with a slightly different reference, you have to handle conversions between them.
At which point you may decide you just want to represent altitude with reference = "same as everywhere else in the program, because if it's the same, it doesn't matter what it is". Which is what most non-GIS code does, except it isn't documented explicitly, and it's up to the user or integrator to ensure the data fed to the program uses the same reference as the user expects to see.