I had this same thing happen to me when I went from 343 to 170. It completely messes with me now going into public and having people actually acknowledge my presence and engage with me. I used to just be a ghost.
All I can remember from my youth was people pounding on me that being fat was basically evil. You see the same sentiment here from time to time so it still exists.