AIUI, you lose energy because the power/torque has to go through an extra differential, transfer case, etc, before reaching the rear wheels.
Going from the engine through the transmission to the front wheels is already a 'fixed cost', but the extra mechanics (and their friction loses) to the rear wheels an 'OpEx' that you don't necessarily have to pay if you don't need to.