- the big boys (Google, Baidu, Facebook etc - basically FAANG) are getting SOTA using private data without explicitly stating it
- their toolkits are too complicated and not suited to non-FAANG-scale datasets/problems
- published results aren't reproducible and academic papers often gloss over important implementation details (e.g which version of CTC loss is used).
- SOTA/leaderboard chasing is bad and there's a general over-reliance on large compute, so non-FAANG inevitably end up overfitting on publicly available datasets (LibriSpeech).
I'm far from an expert, but having spent the last 6 months familiarising myself with the STT landscape, I would say I mostly agree.
First, the author wants more like Mozilla's Common Voice. Not sure if he/she is aware, but I think the M-AILABS speech dataset (https://www.caito.de/2019/01/the-m-ailabs-speech-dataset/) will go a long way to "democratizing" STT.
Second, agree that the toolkits are all complicated and weighted in favour of highly parameterized NNs. This works great if you have FAANG-style data (and hardware), but really isn't pushing the field forward in other respects (compute/training time, parameterization/etc). In all fairness, though, HMM-based frameworks like Kaldi are notoriously complex and take significant effort to wrap your head around. I think speech is just inherently more complex than image or text.
Third - and this applies to ML in general - agree that papers all too often gloss over the important details. I see no reason why publications shouldn't all be accompanied by code and dataset hashes.
I think we're seeing FAANG all converge on a non-optimal solution, simply because we've effectively removed resource constraints. When your company doesn't blink at spending a few million on creating its own datasets, and you have massive TPU farms available at your disposal, there's no incentive to focus on sample- or compute- efficient models.
What's more, they also have the resources to market their successes much more effectively. Think open source frameworks, PR pieces, new product releases, and so on.
This effect is two-fold. First, it creates a lot of noise that crowds out the slower, smaller contributors. Second, it also draws a lot of junior research attention - who would want to risk 4 years of their life on a pathway that might not work when NNs are here today and there's clear evidence that they work (albeit only for certain problems in certain environments with certain datasets).