Most nuclear reactors are thermal power plants and they all need to dump waste heat. Thermal power plants convert heat into electricity with turbines and are limited by dumping the waste heat.

Are you thinking of radioisotope generators? Those aren't reactors. They use thermocouples and need to get rid of the waste heat. The Voyager RTGs have radiator fins.

Thermal power plants don't use up the heat. They make energy through its transfer (like a dam with water).