http://www.newyorker.com/reporting/2009/03/30/090330fa_fact_...
Change it from less than 1 hour of contact per day to something like 4, and I doubt it would be harmful.
Guards are there to keep them in the facility, and more or less from killing each other, but that's about it (and the guards are often as much the ones causing the problems as the prisoners).
Most people don't realize the prisons these days are largely like this: http://www.buzzfeed.com/sgtdopushups/cop-killer-updates-face...
So what I mean is that it's not clear, from a correctional standpoint, that the modern group environment is a safer or more sane environment than the original isolation concept. To my understanding, the main reason prisons have largely moved to a social environment had more to do with cost than perceived harm from extended isolation.
I won't dispute that extended solitary is also mentally damaging...but your idea of more solitary than today for the typical prisoner, and a few hours of contact, is pretty sound to me.