Confucious would reply:

When the wise man builds an incredible machine that can bring us to the Moon, the fool looks at the bags of feces.

More seriously, because of the rocket equation, bringing more payload back would require a lot more fuel. Bringing this additional fuel to the Moon would also require even more fuel.

The Saturn rocket is still the biggest rocket ever made. Probably, the engineers back then did not want to make an even bigger one just to bring back the bags.