The Apify library tries to address most of these issues (I'd say quite successfully). Apify.com provides a platform you can deploy the scrapers on. And yes, at scale.
There's also a software marketplace where you can order custom scrapers. 98% of the projects ran thru it have a 5-star rating (Disclaimer: I moderate that marketplace). Pro-tip: submit your project with a Gmail address to skip sales and reach me directly.