Tricopters are possible as tilt-rotors, where you have two motors spinning opposite directions on mirrored sides, and the third motor is on the midline in front or back and able to tilt left and right. This allows the vehicle to control yaw despite having an inherent yaw imbalance in hover. I suppose you could do any odd number of rotors this way, and more rotors would mean less inherent yaw imbalance if there's only one extra motor spinning a given direction.
Two is possible if the payload hangs below the props, and the props are able to independently tilt.
One requires collective pitch and still at least a small tail rotor to cancel out the yaw, as a helicopter, but that is far more complex and fragile. But it is the most efficient, as fewer larger props are always more efficient than more smaller ones.
So four just turns out to be minimum to have no moving parts besides the props themselves, and still have full control authority, and the control logic is more straightforward than tilt rotors and such.