With the most common North American accent "would've" and "would of" sound very similar. My suspicion is that the error tends to arise in people that have learned language more from speaking than reading.
So in my instance, I would tend to agree with that observation and support that as a large factor from personal experience.