I think you are referring to the [order matching system](https://en.wikipedia.org/wiki/Order_matching_system) of exchanges. This page lists a couple of the common and simple algos used.