My guess is it's designed for fast interactive use, so as long as you do your work in bursts of 300ms or less, you will probably get full speed.
Beyond that, and you'll be power or thermally throttled.
It is more art than science - you want to get the best possible user experience while not using too much power and not exceeding thermal and power limits.
Typically, governors are written as linux kernel modules. They get to choose the clock speed of groups of cores, and which cores are enabled. Some are in trustzone applications or android userland apps though.
Some hardware has extra constraints. For example, the autofocus mechanism on one device I worked on used a lot of power, so while the autofocus was initializing, the CPU had to be scaled down to the slowest speed for a few milliseconds, and the radio transmit disabled. The code ends up a mess because you need to pipe that info from all the other device drivers with no proper interface. A large number of devices do similar shenanigans when the battery is <50% because the battery voltage drops at high loads, and if it's already quite low, then you'll trip a brownout detector and reboot the device.
Thermal constraints are the main one - typically represented as some exponential curves. Ie. you may be at 2.0Ghz for 100ms, 1.8Ghz for 1 second, 1.6Ghz for 5 seconds, 1.4 Ghz for 30 seconds, etc. Depending on the history of the currently running process, the governor would try to decide if it was best to go to 2.0Ghz because this was going to be a quick task, or to only start at 1.6Ghz because it was going to be a long running thing.
Hopefully thats some insight! Check out XDA forums where some devices have a community of people into modding governors for different use cases.