The problem with this argument is that is that it logically dictates that we can only spend money on one thing at a time.
You bring up space travel, but the exact same argument applies to sports, movies, art, technology, evenbthe Internet itself. Once distilled as "how can we spend money on x when y" then it behoves us to identify y, and spend on literally nothing else.
Globally not-spending on defence for example would free up resources enough to substantially raise the standard of living for billions.
On the other hand, spending is raising the standard of living for billions already. Movies spend huge amounts, employing the services of hundreds of professions. The money poured into movies doesn't get destroyed, it gets distributed. (Including globally to places like India, Cape Town, and so on.) So it is with spending on space. The bulk of the money is spent/recycled here on earth.
The problems you highlight (poverty, living conditions, health facilities etc) are complex though and seldom fixed simply with money. In some ways money is the easy part, other factors (like human greed leading to corruption) are trickier.
So, I hear what you are saying, but its not an either-or proposition. There's room for both. I applaud your desire to focus on the problems of others. Don't worry about other areas, focus on the people you can help.