Something like this does exist, production systems rarely use greedy search but have more holistic search algorithms.
An example is Beam Search:https://www.width.ai/post/what-is-beam-search
Essentially we keep a window of probabilities of predicted tokens to improve the final quality of output.