The really good deals will even be site specific. All this can be done via good expertise. With scraping and this type of logic, you'd get blocked quickly from the major sites.
The thing is that if you're looking for a very specific destination, you'll almost never get the best deals. Flight deals work best for people who are very flexible about destination and timing.
When one of us sees Scott's email with attractive deal, we check Google flights for appropriate dates (typically 7-10 days trip) and ping each other to see who is interested in that trip and dates. Then we just book flights typically within 2-6 hours of receiving email. Afterward we research the destination, decide rest of the itinerary for the trip, and book AirBnBs.
We call it travel easter egg hunting.
This is just anecdotal, but if neither Kayak nor Google gets it right enough for me to use them end-to-end, I suspect there's a lot of room for a high-touch, human-intervention kind of searching.