I'm a US citizen (ông Tây) who lived in Saigon (legally) for ~12 years (2001-13) and spent time in Ha Noi and all I can say is that Saigon and Ha Noi are really different.
It seemed that people who loved Ha Noi did not care for Saigon and visa versa.
You seem to be in the Ha Noi group.
Saigon is a worthy place to visit full of people from different parts of Vietnam and around the world. It is culturally unique.