Sounds like me! I'm moving back to CA after ~2 years in Dallas. I didn't hate Dallas, but my fiancee does. But yeah the weather is awful: 115 degree summers, below-freezing winters.
I'm a Texas native. The temps in Texas are nothing like what California gets.
On the other hand, the rockhounding is far, FAR better out here in Cali. But you aren't doing that in the summer. You could do that in Texas in the summer out in the Llano uplift region.