Why would it be exhausted over time? When things come close to the magnet it uses its potential energy to pull them.

When some force acts to remove those objects from the magnetic field, that energy is converted into potential in the magnet again.

Are you claiming that the alignment of the magnet will fade slowly over time when the field is interacted with, more quickly than it would if it were left on a proverbial shelf for millions of years?