Birthers are ridiculous, but you don't actually have to be racist to be a birther. It's driven primarily by partisan politics. Racism just gets mixed in sometimes.
Here's what I see, over and over again: Trump does something that is not exactly racist but ambiguous enough to be interpreted as such. The media goes wild and declares him Hitler. But when you read the details, it's not nearly as bad as the media claim. So Trump gets free publicity and makes his critics look unreasonable. A clear example of this is the "star of David" incident.
It's a sleazy tactic, but very effective. He uses this tactic not just with racism, but with all kinds of manufactured fake scandals.
If you mix in enough of these incidents with your own confirmation bias, you can be very confident that Trump is racist and yet not be able to provide any solid evidence. I suspect this is what is happening here.