Replacement of whatever (necessarily small, otherwise the big city money wouldn't be a problem) industry there is/was with entertainment and tourism isn't the issue. The reality of an economy that is dependent on a specific image or perception (many would say facade) of quaintness is the problem.
You get all sorts of terrible "crab-bucket-esque" feedback loops that quash people's desire to do anything different, innovative or ambitious for fear of harming the golden goose. A large fraction of the people who actually want to do things (boring old normal business endavours that underpin a sustainable economy) leave because whatever their trade is doing it elsewhere and not putting up with all the extra overhead crap of doing it in a tourism/entertainment economy. The fact that a much larg(er than in a "normal economy) portion of the customers any given business sees will only ever transact with them ones creates a bunch of perverse incentives to do dishonest business.
It's like everything that's wrong with a college town combined with everything that's wrong with a wealthy retirement community.
Edit: strike CT from that list of examples, I was going for a "Cape cod but in the NYC economic region" and that part of CT is more like NYC's wealthy NJ suburbs.