I'm a supporter of solar, and trying to get it on my house.
But I also did some back of the envelope calculations that showed, just if we had enough Powerwalls to backup US peak demand for one hour it would require 10x the global annual mining production of lithium. And that's just one hour. And that doesn't include the electricity production.
It's generally estimated that US power, with good transmission, would require enough solar panels to cover the entire state of Massachusetts. I think you're right that it isn't the land cover that would have much effect, after all buildings and roads already cover a lot of land. I think it's sheer material production.
Mining is almost entirely powered by fossils, it has to be. And so is most transport. And so is recycling of metals. So the energy density of an energy source really is a zero sum game. If it takes a millionth the material for one source versus the other, that adds up.
Then in maintenance, solar farms are truly "farms"- they require a lot of water to wash away dust to operate optimally. A states' worth of water is significant.
Then in recycling at end of life, and this is why I got so excited about nuclear as a somewhat hippie child growing up around oil companies in Oklahoma, solar is going to require a lot of energy to recycle, while nuclear can produce energy in recycling its fuel.
The main reality check, to me, is: what is the energy density of this energy, and if emitting, how much pollution? Coal is far more energy dense than wind, which is why humans evolved from windmills and wood to coal. But it's so polluting which is why we are all working towards better sources.