So a speculator, they are taking a chance that the work of others will improve the community and raise their investment. Got it, morally not right, but also very risky, these people lose their shirts just as often as they make it big.
Ok, then the old couple who bought the "shack" in the 1970s and finally are selling now. Why do you claim they weren't a contributor in the community? Let's say the guy was a garbage man while their wife worked for the DMV. They raised their kids and whatever, in that shack. They never banked on it being worth $2MM, it just happened. They provided value to the community, ok, they didn't create that much value, but its not like they were camping on the land just for the appreciation.
The real problem with CA is proposition 13, which makes the cost of owning and sitting on land relatively cheap over a long period of time. But ya, that garbage man would have been forced out of their house maybe in the 90s without it, but it would make property speculation a lot less profitable because long term positions would be very expensive.
In China, this is even a larger problem because they lack property tax at all in most places, you pay your taxes up front in a land acquisition tax and then you can just leave your apartment unrenovated and market it as brand new 20 years later to make a few million dollars.