Primary energy for solar panels is the sunlight hitting them not the electricity they produce. Fossil fuels care about primary energy because you pay for the fuel. Further, roughly 40% of global shipping by weight is moving fossil fuels.
> solar power plants have a capacity factor around 20%
This varies widely by technology and location. ~30% isn’t uncommon in the US. Capacity factor 29.7% (average 2015–2017, MS1) https://en.wikipedia.org/wiki/Mount_Signal_Solar tracking vs fixed makes a huge difference. Rooftop solar sucks vs grid scale in terms of production but you save on grid transmission losses the numbers aren’t directly equivalent.
Further PV panels degrade every year, but you see 30 year old panels still in use.
> human energy consumption to continue to grow > 20 years from now we might need four times that much
Population growth + Electric cars + AI changed US electricity consumption from 4.03TWh in 2018 to 4.19 TWH in 2025. That’s less than 0.6% per year, suggesting 4x increase in primary energy consumption from AI in 20 years as a meaningful possibility is insane. https://statbase.org/data/usa-electricity-consumption/