It's often bemoaned that personal cars are a waste of money and space: they sit idle most of the time, we have to build parking lots to accommodate them and that takes space. It would be so much better if we could just hire cars on demand! But this doesn't scale because the demand is not spread out during the day -- absent carpooling, you still need one car per person twice a day, meaning the supposed savings don't exist.
The same could be said for housing. If I'm at work all day, then my house is sitting idle (it's not housing anyone) and it takes up space. Why not let someone live in my house while I'm at work? This also doesn't scale but for a different reason: we all work during the day (excluding night-shift workers) so at night we still need one house per person (ignoring families, stretching the metaphor a bit).
Now add "vacation homes" into the mix, now you're looking at ~20% of housing (calculated from empirical analysis) in any vacation hot-spots being rental properties. This means that 20% of the existing housing that could be available for permanent residents is missing.
Saying "just build more housing" is just not realistic. 20% more housing is a ludicrous amount of housing!
Hotels, on the whole, are much more economical in space utilization because people who are going on vacation do not vacation to _visit a home_. They go to do things in the place they are visiting and are thus _not at home_.