I haven’t read the book, but my understanding is that racism did play a preferential factor in that an ethic group that achieved geopolitical dominance tended to impose sanctions on more economically primitive people both intentionally and unintentionally. An unintentional example was the case with 1000 years of exposure to many pandemics for Eurasian people that isolated people never experienced, but once exposed suddenly dwindled.
Chinese and European technology was roughly equivalent at the time, but the huge increase in economic activity by European states associated with colonisation gave them a lead that was impossible to reverse.
And yes, the book make it extremely clear how little race had with anything.
A more interesting comparison is European conquerors to civilizations of the Americas during the colonial era. The Aztecs were only conquered as a result of Spanish exploiting social unrest. They didn’t have the numbers themselves for their technological military advantage to matter, though that wasn’t the case with the Incas further south. Had the Incas developed metal working as advanced as their agricultural refinements they probably could have repelled the Spanish invasion. Conversely, the Spanish were never able to conquer or subdue the Maya communities around Mexico’s current southern border due to the environment, geography, and decentralization.
You rightly point out that there has been tremendous exploitation of (for instance) Africa by Europe, and that a lot of this exploitation has been justified by racism.
But separately from that, consider the question: why were Europeans able to do this to Africans in the first place?
If you asked question back in the Nazi regime, you'd probably get answers like, "Europeans are somehow inherently racially superior to Africans, and this superiority allowed them to become dominant."
Guns, Germs, and Steel lays out a different and detailed explanation. The author is quite aware of this history of racist explanations. He provides a competing theory, backed by considerable research and evidence. I found it quite good, but also very dry. :)