"Basically the wealthier folks don’t like the aesthetics"
That's not inherent though. A nice French brocante sale basically sells tat, but its nice and acceptable.
I don't think its impossible that this flea market could become a hipster hang out. One of the eateries starts making a name for itself, and people start seeking it out. It isn't just poor people that want a functioning high st.
Edit: Spelling