You make a very good point. However, the obsession with economic growth itself isn't the problem - it's how economic growth is measured. As an example, if a hurricane sweeps through Florida doing billions in damage, Florida is surely worse off, but because of the rebuilding activity that follows, it will register significant growth.
As you state, we need to think about what makes people happy. Economics is a great tool to do that, but it will require a rethinking of how the economy is measured.