The simple problem is that 99.999999999999999999% of self-professed libertarians I've encountered who simply say "the government should get out of the marriage business" only ever say that. They never lobby for (or introduce, if they're in a position to do so) bills to actually achieve that end, and in fact usually do nothing and let the status quo -- which until very recently was "keep the discriminatory laws on the books" -- stand.

So in extensional terms, such "libertarians" are often indistinguishable from theocrats on key social issues (i.e., regardless of what different set of inner philosophical principles they might claim to be following, the outward result in terms of lawmaking is identical). Which in turn means they're not going to get my vote.