If a skilled surgeon leaves Nigeria and migrates to Europe or the USA - their hometown is now worse off, because they lost a good specialist and access to healthcare now became worse. And whatever remittances that person sends back home don't offset this.
Rich Western countries poaching skilled workers from poor countries make those countries even poorer.