> “code” has become a commodity: many powerful ML models are open source today. The real challenge is to grow and curate quality data.
The main recent improvements in translation and speech recognition were all new mathematical methods that enable us to use uncurated random data and still get good results. CTC loss allows to use un-aligned text as groundtruth. wav2vec allows to use random sound without text for pre-training. OPUS is basically a processing pipeline for generating bilingual corpora. Word embeddings allow to use random monolingual text for pre-training. We've also seen a lot of one-shot or zero-shot methods. Plus XLS-R was all about transfer learning to reuse knowledge from cheap abundant data for resource-constrained environments.
My prediction for the future of ML would be that we'll soon need so little training data that a single person can label it on a weekend.
On the other hand, I know first-hand that almost nobody can use the "open source" ML model and deploy it effectively in production. Doing so requires sparsity, quantization, conversions, and in many cases l33t C++ skillz to implement optimized fused SSE ops so that a model will run with decent speed on cheap CPU hardware rather than mandating expensive GPU servers.