Improving a Machine Learning System – Broken Abstractions

Most ML solutions in most businesses have either questionable business value or debatable ROI. This lends itself to many products which are shipped quickly and then semi abandoned. In many cases the ML that's shipped barely outperforms the heuristic solution if at all.

Further, most of the teams tasked with maintaining or improving these systems have little practical interest in the production service - it's much easier and rewarding to bemoan a slower portion of the business while publishing research than improve the production service. It's not uncommon to encounter ML teams who claim multiple year's worth of innovations are backlogged against engineering. I once encountered a 20 person team who owned the core bidding algorithm for a large ad-tech startup which hadn't made a change in 3 years. The business believed that they were making regular releases.

What this all says is that we’re in a strange valley of ML maturity where enough people know about it to try it out, but few are able to navigate how to make a sustainable investment.