The "physical limits" of renewables are not limits to our current energy use or even an order of magnitude more. Similarly, the amount of materials is still far far far less than what is required with fossil fuels. And the amount of material needed to build a 1GW nuclear reactor is within a factor of 5-10 of what it takes to build an equivalent amount of solar power, last time I checked. Both of these are far far far less material than what's needed to generate equivalent energy from fossil fuels, it doesn't even really compare.
The complaints against renewables are pretty much insignificant, and definitet not applicable to reality, but these are the complaints that get perpetuated and listened to again and again in utility decision makers. Meanwhile, these same decision makers are not avoiding nuclear because of what environmental groups are saying, they are avoiding it only because of financial reasons. What it took to restart nuclear construction wasn't protection against environmenta lawsuits, it took the South Carolina and Georgia legislatures passing bills that would allow the utilities to bill for construction even if it weren't overbudget, and even if construction failed and the project never generated electricity. Cost is the primary barrier to new nuclear, not environmental opposition.
Nuclear has spent 50 years over-promising and under-delivering, solar and wind have spent 50 years under-promising and over-delivering. They are a study in contrasts between a hype driven field (nuclear) and a results driven field (renewables).