If I understand correctly, this sounds like an idea that I've been thinking about for a little while.

Essentially, you do binary search-insertion into a list, where the comparison function is a prompt to the user asking "Is A better than B?" (If it's too difficult to judge "betterness" between two items, you could just as easily swap in a different comparison. "Is A funnier than B?")

One thing that people always ask when I mention this is: "What if A and B are equal?" Well, then you answer no, because A is not better than B. If your answers are consistent, then A and B will end up next to each other in the list.