Why don't sources try to measure the frequency response of the driver + cable + headphone circuit and apply a digital prefiltering step to compensate these distortions?
The best thing is for the speaker manufacturer to assume zero ohm output impedance from every amplifier, and for them to do their work to get their speaker to have a close to flat response. (Or whatever: some markets prefer a huge bass.) Then it has that designed response in any such amplifier.
Because the effects are negligible and trivial to overcome with even basic attention to design.
The distortion of speakers or headphones is orders of magnitude higher than the distortion from even a basic headphone amp design in the modern era.