Yeah, I tend to agree with you here. The difference I see is that solar has already been heavily invested in and developed recently, so it's probably near the limit of today's material science.
My personal philosophy about technological progress is based heavily on the idea that material science is the principle barrier to what we can accomplish. That's why humanity's progress is recounted in "ages" named by material.
So, I think solar panels can get maybe 2-10x more economical, whereas other options like new hydrocarbon sources, fission, and fusion can probably all get at least 10x more economical than the maximum possible with solar, purely due to the physical limitations of the technology.