Currently, the process is that you need to find a company that is willing to sponsor you, then that company needs to file paperwork with the government to explain the wages, job description, etc. And then this application is either accepted or rejected based on whatever guidelines they use[1].
If we keep this situation and start allow workers to move around wherever they want, the company just ate the paperwork legal costs and the role they hired for might not even stay filled for long, prompting them to redo the process.
[1] https://www.immi-usa.com/h1b-application-process-step-by-ste...