You also get global dust storms enveloping the planet every few years for several weeks at a time. These leave a lot of dust on panels, but they also drastically cut the solar radiation received at the surface.
Solar panels degrade faster on Mars too. You could replace them by sending new ones, but that will add up to a lot of mass. Or you could design panels that degrade more slowly, but that's not a given yet.
Basically solar is not going to scale well on Mars. It might work for a small facility, but not for a city.