Birtherism was inherently racist in and of itself, though. If Obama was white it simply wouldn’t have been a factor. It relied on suspicion of an “other” as a key component.
You don’t lynch someone in effigy because you’re trying to say that you don’t think they aren’t 35.
It is known and not under any debate that John McCain was not born on US soil (he was born on an army base in Panama), and yet Obama’s legitimacy was the one in question. It’s very obvious why one candidate was considered a foreign other and one was not.