Our family does about 80% of our grocery shopping at Aldi despite having far above median income. The number of Teslas and higher end SUVs I see in the parking lot tells me I’m probably not the only one.
I'm going to say you are in the minority. I remember many people near me growing up, who weren't very well off themselves, who considered Aldi to be the "cheap store all the foreigners go to". Regardless of how one feels about the perception, the perception certainly exists.