I don't think Sam Altman would disagree with this statement. He also recently led a $500 million investment in Helion (https://www.helionenergy.com/articles/helion-raises-500m/), with the goal of achieving net-zero fusion energy by 2024.
Also, a lot of major technological advances enjoyed by the masses today started off as niche defense projects (the Internet, GPS, modern flight, etc). If we're able to prove out the technology for the niche defense use-case, it's not out of the realm of possibility that it will eventually be worth investing in developments of the same technology in ways that may benefit the masses.
That said, it is true that Sam Altman does not hold a strict pacifist view on technological improvements in defense (i.e. funding of killing machines). That's a thornier debate.