My wife is Panamanian and having just recently traveled there I got the same impression. I was asking everyone about it and they held the same opinions to yourself.
It's also worth noting that the reason the canal was put in Panama over Nicaragua was due to earthquake activity which Panama did not have.