SEO isn’t magic. Google decided to deprioritize Wikipedia for example which noticeably degraded results. Where major websites show up has nothing to do with SEO and simply relates to what they think is important.
I remember using an open-source ML model (trained on Buzzfeed article titles) to detect YouTube clickbait based on titles and it worked brilliantly, and that was just downloading some code on GitHub and running it as-is. I'm sure the same could be applied to search results and you could achieve much better quality if you actually put some effort into it.
I very much doubt this is some kind of hard problem as opposed to Google just giving up because their business model doesn't actually incentivize good search results.
Link to the Repo please?
Google could start incentivizing high quality unique content and websites from domain experts, but they have decided they can’t make as much money off of independent publishers as they can from marketing/content farms.