The energy comes from the incredible magnetic force spent to align the atoms when the magnet was created, and it will be slowly exhausted over time.
When some force acts to remove those objects from the magnetic field, that energy is converted into potential in the magnet again.
Are you claiming that the alignment of the magnet will fade slowly over time when the field is interacted with, more quickly than it would if it were left on a proverbial shelf for millions of years?
This is true, it's just not why magnets can lift things.