parnoux

Interesting that a simple model and hand crafted features would perform this well

Yes, that's correct. We integrate with the major ML frameworks to monitor serving data and compare it to the training data to identify potential error patterns and mine the live stream for data to fix them. I'd love to show you the product in more details and get your feedback if you're open to it !

Thanks for sharing

Thanks for the question and the papers. Like some of those companies, we are believers in the data centric approach to ML. But labeling is not our focus (unlike Snorkel, HumanLoop, Prodigy or Cord.Tech). We focus on model diagnosis and mining the best data to improve them. So there are more similarities with Aquarium or Lightly.

There is a great talk from Tesla [1] on what they call the "Data Engine" (that probably inspired some of us :)). One of the things we took from it was that in order to truly close the loop on the ML data flywheel, we needed to turn production into a reliable datasource. It had to become accessible, understandable and minable. To achieve this we took the approach of combining ML observability with Active Learning mining frameworks. Combining both is important in our view because Observability tells how the model behaves in the real world and Active Learning finds the right samples to fix / improve the model on real world data. They go hand in hand.

Technically, it means that we integrate with serving and labeling platforms. We ingest data both in streaming and batch. We can mine on production streams including on device (for iot use-cases where accessing data is a challenge). We have an extensive set of metrics to understand model behavior in the wild and solve use cases like data drift (detecting it, triggering mining and sending the data for labeling/retraining). And we are geared toward automation.

Regarding which data sampling works well and which doesn't, we found that it's not a one size fits all. Combining uncertainty sampling and diversity sampling is very powerful in a lot of use-cases and can perform as good as random sampling with 10x less data some times. But model based sampling strategies can also underperform on drifted datasets (essentially a model can be very confidently wrong on new kind of sample), hence the need to also have similarity sampling techniques.

Overall, we were able to show that we can intentionally drive specific model performance metrics, either globally or locally, by picking one technique vs another. Happy to share more if you want.

[1] https://www.youtube.com/watch?v=Ucp0TTmvqOE&t=7714s

In the example shown in the product tour, we use an approach based on Diversity Sampling. Basically we look for the datapoints that would be the most representative of the drifted domain (and therefore be outliers to the training domain). Here is a blog post that summarizes some of those techniques [1]. I also found this paper that describes a similar approach [2]. Happy to chat more about it.

[1]: https://towardsdatascience.com/https-towardsdatascience-com-... [2]: https://arxiv.org/abs/1904.03122

I don't know how tight your legal restrictions are but we work from metadata only. We don't need your text / img / audio. We just need their embeddings and a few other stuff. And we are working on a self hosted version as well. Out of curiosity, what would you expect in terms of pricing ?

I don't think our assumptions are so far appart. The methods you mentioned made it from research to the open source community fairly quickly. In fact, most companies rely on this kind of open research to develop their models. In a lot of use cases, it has become more about finding the right data that improving the model code. (I like Andrew Ng thoughts on this: https://datacentricai.org/) At the same time, there are still a lot of unsolved engineering challenges with the code when it comes to productionalizing models, especially for real time speech transcription.

And we agree with your prediction. That's why we started Dioptra: to come up with a systematic way to curate high quality data so you can annotate just the data that matters.

You are right. Being able to learn good feature representations through SSL is very powerful. We leverage such representation to perform tasks like semantic search to tackle problems like long tail sampling. We have seen pretty good results mining for edge cases. Let me know if you'd like to chat about it.