Google trends would disagree with you there [0].
Anecdotally, I've only ever heard the name Fermi in the phrase Fermi Paradox. Oppenheimer I know much more about, and I only know there's a movie about him because it's all over HN this month.
I'm sure that the red scare did tarnish his name for decades, but it looks to me like those decades had ended by 2004 when Google's trends pick up.
[0] https://trends.google.com/trends/explore?date=all&geo=US&q=%...