People don't care that you're there. They care that you are making boutique $15 sandwich shop(s) a viable business where they were not previously causing rent to go up driving the burger joint they loved out of business or whatever.
Ideally you've retire to Arizona or Florida or some other traditional retirement or tourism destination with an economy has had an influx of money for a long time and reached equilibrium. In lieu of that the next best thing is assimilating and living like the locals.