razcle

Hi NMCA, I'm Raza one of the founders of Humanloop. I totally agree that transfer learning is one of the best strategies for data efficiency and it's pretty common to see people start from large pre-trained models like BERT. Active learning then provides an additional benefit, especially when labels are expensive. For example we've worked with teams where lawyers have to annotate.

In terms of breakthroughs in recent years, some things I'd point to would BALD (https://arxiv.org/abs/1112.5745) and its applications in deep learning as well. There has also been progress in coreset methods (https://openreview.net/forum?id=H1aIuk-RW).

I think that you're right that it used to be much to hard to get active learning to work. Part of what we're trying to do is make it easy enough that its worth the benefits.

Humanloop | Infrastructure for AI | Backed by YC and Index | London + Remote Hiring

- Software Engineers: full stack and front-end

- Machine Learning Engineer

(see jobs.humanloop.com for full details)

We're a team of ML researchers and Engineers who've worked at Google, Amazon and Microsoft research on some of the biggest ML projects out there.

ML and deep learning are a new software paradigm that needs new tools. We're building a platform for Human-in-the-loop ML that drastically reduces data needs and accelerates time to deployment. In the future people will program by teaching and curating datasets. (https://medium.com/@karpathy/software-2-0-a64152b37c35), we're making software 2.0 possible.

Team: humanloop.com/about

Contact the founders at founders@humanloop.com

Humanloop | Infrastructure for AI | Backed by YC and Index | London + Remote

Hiring

- Software Engineers: full stack and front-end

- Machine Learning Engineer

(see jobs.humanloop.com for full details)

We're a team of ML researchers and Engineers who've worked at Google, Amazon and Microsoft research on some of the biggest ML projects out there.

ML and deep learning are a new software paradigm that needs new tools. We're building a platform for Human-in-the-loop ML that drastically reduces data needs and accelerates time to deployment. In the future people will program by teaching and curating datasets. (https://medium.com/@karpathy/software-2-0-a64152b37c35), we're making software 2.0 possible.

Team: humanloop.com/about

Contact the founders at founders@humanloop.com

Hi, sorry I missed this is earlier! We're in many ways complementary to weak labelling techniques (like Snorkel) and are actually working to include them in the tool. We think weak labelling is a great way to overcome cold starts and active learning then helps you improve rapidly.

The big difference between us and Snorkel is our emphasis on active learning and HITL deployment. We think the existing paradigm of ML deployment is very waterfall and slow.

Would love to hear about the annotation interface your building. Agree there should be one place with all those features! (We're hoping it will be Humanloop ;) ).

At the moment we dont support the ability to render the HTML but it is something that has come up before. One of the teams we're speaking to wants to classify blog posts and would like to be able to preserve their formatting. If this is something that's important to you we would consider adding it so maybe drop me an email at raza[at]humanloop.com and we can discuss?

Thanks! Prodigy is a good tool and we definitely were inspired by some of their UX decisions. Reducing each decision to a small atomic unit and avoiding context switching makes a lot of sense.

Our starting place is similar to Prodigy in that we also see active learning as a key piece of the puzzle but we think to make active learning work reliably really does need taking into account parameter uncertainty. As far as I know Prodigy doesn't do this. We are also working to make our active learning work at the level of batches and be cost-aware. Often the most valuable examples to label for the model are the most time consuming for humans and we work to trade this off.

A few other differences are that we do offer a cloud hosted solution so getting set up is much faster and it's more natural for us to be able to accomodate team annotation and quality assurance. By providing a hosted model we also give you the option of deploying features very quickly and continuing to improve them post deployment.

I'd be curious to know the barriers that Saas introduces for you?

The model you train and data you upload are yours to own, unique to you and don't get shared across users or reused for any other tasks so hopefully you shouln't feel too much like your building our IP ;)

In terms of deployment options, we're trying to lead with cloud hosting by default but know that for a lot of people the whole reason they're annotating in house is privacy so we've been exploring deploying in your VPC and for larger enterprises on-prem.

Interested to hear more of your thoughts on the pricing model, this is something we're still iterating on so I'd be interested what you think would be most compatible with your use cases?

We wrap a lot of popular frameworks and have implementations of most SOTA models. By default we use a multilingual BERT model so it should work out of the box on different languages.

Hi,

Raza here (one of the other co-founders). Good question! I think our visions are quite different even if our starting points look similar.

Scale has always positioned themselves as an API to human labour and their goal is to abstract the labelling task away from the end user as much as possible. So scale works really well when you can easily outsource your annotation task.

Our ultimate goal is to try and give domain experts the ability to teach ML models themselves. We're much more focussed on NLP and on tasks that require domain expertise and are hard to outsource. For people where deep domain expertise matters or their are privacy concerns, Scale isn't really an option and we're building tools for them.

On another point, Scale makes its money by charging per annotation so we think they aren't as incentivised to reduce how much you need to label.

thanks!