Downsides include cost of fuel with low fuel economy, difficulty manuvering and finding parking in dense locations, and potential cost of RV hookups. Plus an RV is a vehicle which comes with its own maintenance issues, and I'd imagine insurance is likely high relative to a car. They are expensive, $200,000+ for something that's large enough to comfortably live in.
Does anyone have experience doing this, or know someone who has? Upsides? Downsides?