Solar is like 10% of electric production, which is about 20% of energy use.
So if you expect solar to replace most of that, at steady state you probably need more panels per year than are currently being produced. No reason to expect diminishing production to increase costs vs today (they might rise vs the eventual floor).
I agree that we have an enormous calorie surplus that we can turn the knob on.