This isn't a philosophical debate. This is an empirical debate. Non-competes have both pros and cons, and empirical studies suggest the pros are exaggerated and cons under appreciated. They're also not the only tool in the toolbox. For example, there are NDAs, trade secrets, and tortious interference, among others. Compensation and other incentives can be restructured.
Finally, your scenario only begs the question: so what? For the sake of argument, let's assume that it's somehow unfair to the original employer. What about the customers? Why is it fair that they've lost the option of working with a guy they like, probably for less money? There are two sides to the coin. That's why I said it's an empirical question: we can work out which option produces the better outcome overall, and the facts strongly suggest the best outcome is the world where non-competes are generally unenforceable.
One of the biggest and last remaining areas of productivity growth in modern service economies is putting people to their most productive task. There are huge structural inefficiencies that result in person doing task A when they'd be much more productive doing task B. This is much more of a problem in a service economy than a manufacturing economy. And one source of this inefficiency, and an increasingly common source at that, is non-compete agreements.
That doesn't seem to happen in California where noncompetes are banned. This is one of those things that seems like it might be a problem but doesn't actually happen very often.
Hell, there's case law that a myspace friend list can be a trade secret under the right circumstances.
https://en.wikipedia.org/wiki/Ross_Perot
> After he left the Navy in 1957, Perot became a salesman for IBM. He quickly became a top employee (one year, he fulfilled his annual sales quota in a mere two weeks)[10] and tried to pitch his ideas to supervisors, who largely ignored him.[11] He left IBM in 1962 to found Electronic Data Systems (EDS) in Dallas, Texas, and courted large corporations for his data processing services. Perot was refused 77 times before he was given his first contract. EDS received lucrative contracts from the U.S. government in the 1960s, computerizing Medicare records. EDS went public in 1968 and the stock price rose from $16 a share to $160 within days. Fortune called Perot the "fastest, richest Texan" in a 1968 cover story.[12] In 1984 General Motors bought controlling interest in EDS for $2.4 billion.
I don't see a reason why we should be okay with removing such powerful leverage from the workers hands.