Going down this road, you are about to discover that visual weights, slopes, and assorted other minutia also differ, leading inexorably to the ransom note effect. Some fonts are designed to work together; those that aren't, don't.
Visual height is the tip of the iceberg.