Privacy advocates are not a subset of libertarians. Libertarians may be privacy advocates, but there are many privacy advocates that are not libertarians.

Privacy advocacy is popular among the upper classes, because they really don't want anyone from the lower classes sticking their noses into their business.

Libertarians are unpopular because some of their policies are deeply unpopular. For example, many libertarians oppose the Civil Rights Act of 1964 because they feel it violates their first amendment rights.