No, package formats aren't just different sets of instructions, because on each distribution certain files might land on different places.
Plus someone has to keep track of those instructions for every single distribution.
Finally, supporting the same format isn't enough, for example a RPM for SuSE isn't the same as a RPM for Red-Hat.
Well, apparently you forgot there are people still using ALSA and OSS.
When doing desktop applications anyone that cares about UI/UX of the respective users wants to integrate with the menu system, notification on the toolbar, context menus, drag-and-drop of the window manager, printing,....
So it isn't just components.
Of course, if the goal is to have a plain twm experience, then forget about what I am saying.