So some people sincerely believe Yarvin's very presence is a threat while others find his online philosophical rants to be little more than meanderings.
With that said here's my answer to your thought experiment:
Yarvin would be uninvited if he had been proven (not alleged) to have:
* harmed or advocated harm to anyone * rallied for pro-discriminatory policies * lobbied for pro-discriminatory policies * addressed someone with discriminatory slurs
If he were to deny the evidence then it would be a done deal and he would be banned.
However, everyone gets a chance to repent in my book. So if he had been proven to do these things and he came clean (or eventually came clean) then he would get a chance to publicly apologize online for wide distribution - and again live at the conference at the start of his talk.
---
The real answer to your question is that I would not associate with anyone proven to have done abhorrent things. But I regularly associate with people who have all kinds of abhorrent thoughts.