Sodomy, he argued, was not just harmless but evidently pleasurable to its participants. The mere fact that the custom was abhorrent to the majority of the community no more justified the persecution of sodomites than it did the killing of Jews, heretics, smokers, or people who ate oysters – "to destroy a man there should certainly be some better reason than mere dislike to his Taste, let that dislike be ever so strong".
It's a shame that Jeremy Bentham's name is nowadays more associated with Lost than the utilitarianism he pioneered. An Introduction to the Principles of Morals and Legislation is a good intro to his general line of thinking: http://oll.libertyfund.org/titles/278