Not just energy density, it's also a stability issue. A significant chunk of energy usage is constant (base load), and there's also a fraction that varies over time as everyone comes home after work and turns on their air conditioners and ovens (peak load).
Base load is ideally handled by an energy-dense approach that produces a very stable level of power. Peak load can be handled with more variable sources of energy, but at the end of the day these are usually backed up by gas-fired plants that can be quickly spooled up to match demand.
The problem with photovoltaic solar is that its capacity undergoes a significant level of variance on both an instantaneous and a medium-term scale (time of day). As such it's best treated as a peak source. Otherwise you need enormous, enormous batteries to try and smooth the load. Using electric car batteries and so on is a good idea from the perspective of the utility, but it's a raw deal for the consumer. The lifetime of a battery is limited by charge/discharge cycles as well as time, and they're stealing your cycles. A replacement battery is an enormous chunk of change, it's about the same cost as replacing an engine.
Note that this is not true of all solar techniques. For example, there's a solar-concentration technique that stores heat in molten salt, which helps to smooth out short-to-medium variation. When you lose the sun, you get a smooth curve that deteriorates over a period of hours to days rather than an instant dropoff. This is suitable for use as a base source.