I think people just don't want to deal with slavery and race bullshit in their fantasy setting, which I understand.
So it's just annoying and tone deaf and doesn't add anything, plus the fact they are simian and transported on tall ships and auctioned off as slaves.
If you can't see why someone would would say "wow, a slave monkey race transported on ships. That sucks." Then you aren't arguing in good faith. It is not that much of a stretch to see it as a little demeaning.
Depictions od people of color as animalistic is undeniably a thing people used to do to be demeaning to them.
So I would argue you are having a failing of imagination.