Agreed, although a midi controller by its generic nature does have to adapt to very different hardware, so that for example the knobs, switches etc. placement doesn't reflect functionally the hardware it is connected to, which sometimes would help immensely to ease the operators work.
I wish there was a controller made of pluggable micro modules, that is, say 2x2 cm modules receptacles that could host a potentiometer, a pushbutton, a few leds, a sensor pad etc. where no matter where the module is placed, it will link to the backplane processor getting its identification number and function, then the user can move them around to replicate the functional arrangement of an instrument panel to make programming much easier.
It's more or less the same idea I'd like to develop for computer keyboards, where pluggable keys (pots, haptic transducers and sensors etc.) could be moved around still keeping their programmed function. With todays technology it wouldn't be so hard to develop; very likely the most expensive part would be the plugs/receptacles hardware rather than their digital hardware or firmware.