I believe you're thinking of Yao's Millionaires' problem [0].

[0] https://en.wikipedia.org/wiki/Yao%27s_Millionaires%27_proble...