I have also heard that prefab is really common - in fact just walking around Tokyo it does look like there's a ton of prefab houses. That said I've toured around some and been inside some, and they were very comfortable, more so than my current rickety San Francisco apartment and similarly to brand new NYC ones I've been in - so I don't consider them lower quality than what I typically see in American cities just because they're prefab. Rather it's far more efficient and that keeps costs down.
There is something to be said about traditional Japanese living being more space-efficient - folding out your futon means that you don't have to have separate permanent bedrooms, dining rooms etc like a Western apartment would have, which allows for you to have a decent living in ~220 sq ft like I did - but if you want to live like a Westerner, you will have to pay up for that, and I do hear that Western style apartments are more popular these days.
After living in San Francisco, I felt like I had hit the jackpot, but as it turns out that's just "normal" in a functioning city.