306 karma · joined January 18, 2020
I think HN knows that anyone can prompt LLMs. I do think its interesting though that this has allowed PMs/SMEs to direclty influence products that are deployed to millions of people. That seems genuinely novel. Maybe I over egged it
wrt did you read the article? I was quite specific about the ways I think LLMs are blurring the lines. I don't think its true for general engineering but I do think its true for applications being built with LLMs.
Also its still very early
But I guess I need to up my game if you can't tell the difference
I totally agree that we're not at a point where AI can write most code. Though, I didn't ever say that. I just think its blurring the boundary between engineers and PMs with both taking on more of the others role.
Also, it shouldn't be surprising that the product we're building is aligned with what we believe about the world :)
R
OP here. Thanks for the (harsh!) feedback, I'll take it in a growth mindset.
The post does genuinely reflect my experiences and I do believe what I said.How would you advise I change the post to make it better?
Which parts do you think are untrue?
Thanks!
I think this is reasonable. Giving researchers access is great but for most small companies they're likely better off having a service provider manage inference for them rather than navigate the infra challenge.
I know that HN likes to nerd out over technical details so thought I’d share a bit more on how we aggregate the noisy labels to clean them up.
At the moment we use the great Skweak [1] open source library to do this. Skweak uses an HMM to infer the most likely unobserved label given the evidence of the votes from each of the labelling functions.
This whole strategy of first training a label model and then training a neural net was pioneered by Snorkel. We’ve used this approach for now but we actually think there are big opportunities for improvement.
We’re working on an end-to-end approach that de-noises the labelling function and trains the model at the same time. So far we’ve seen improvements on the standard benchmarks [2] and are planning to submit to Neurips.
R
[1]: Skweak package: https://github.com/NorskRegnesentral/skweak [2] Wrench benchmark: https://arxiv.org/abs/2109.11377
- Software Engineers: front-end specialist
- Machine Learning Engineer
- Interaction designer
(see jobs.humanloop.com for full details)
We're a team of ML researchers and Engineers who've worked at Google, Amazon and Microsoft research on some of the biggest ML projects out there.
ML and deep learning are a new software paradigm that needs new tools. We're building a platform for Human-in-the-loop ML that drastically reduces data needs and accelerates time to deployment. In the future people will program by teaching and curating datasets. (https://medium.com/@karpathy/software-2-0-a64152b37c35), we're making software 2.0 possible.
Team: humanloop.com/about
Contact the founders at founders@humanloop.com
I think the mistake of this quote is in the application of the expertise. The bitter lesson is that data + compute can outperform inductive biases but that doesn't mean you don't need domain expertise to get the right data.
Almost all data science workflows treat the annotators or subject matter experts as secondary. The tooling isn't set up to put them at the centre of the process and make it easy for them to collaborate with the more technical folks.
Perhaps it should be obvious but its definitely over looked in much of academic ML and in MLops.
One of the main differences is that we've pretty exclusively focussed on language rather than vision which has quite a different tech stack.
We also view human-in-the-loop not just as a way to get better data but actually as a better deployment paradigm.
P.s You're right that David is awesome btw!
You seem to be pretty clued up on the area, what do you see as the pros and cons of an end-to-end approach?
Hiring
- Software Engineers: full stack and front-end
- Machine Learning Engineer
(see jobs.humanloop.com for full details)
We're a team of ML researchers and Engineers who've worked at Google, Amazon and Microsoft research on some of the biggest ML projects out there.
ML and deep learning are a new software paradigm that needs new tools. We're building a platform for Human-in-the-loop ML that drastically reduces data needs and accelerates time to deployment. In the future people will program by teaching and curating datasets. (https://medium.com/@karpathy/software-2-0-a64152b37c35), we're making software 2.0 possible.
Team: humanloop.com/about
Contact the founders at founders@humanloop.com
We're still trialing the Hugging Face models in beta but would appreciate your thoughts and feedback.
I may update the post. thanks!
Synthetic data is particularly valuable when even the unlabelled data is expensive to obtain. For example if you want to train a driverless car, you may never see an ambulance driving at night in the rain even if you drive for thousands of miles. In that case, being able to synthesise data makes a lot of sense and we have lots of tools for computer graphics that make this easy.
Synthetic data can also be useful to share data when there are privacy concerns but my own feeling here is that there are better approaches to privacy preservation, like federated learning and learning via randomised response (https://arxiv.org/pdf/2001.04942.pdf).
In general though, outside of some vision applications, I'm pretty sceptical of synthetic data. For synthetic data to work well, you need a really good class conditional generator. E.g "generate a tweet that is a negative sentiment review" but if you have a sufficiently good model to do this, then you can probably use that model to solve your classification task anyway.
For most settings, I think synthetic data will work for data augmentation as a regulariser but will not be a substitute for all labelled data.
For the labelled data, active learning should still help.
Entropy selection for pool based methods looks at the output probability for each prediction of the model in the unlabelled data-set. Then it calculates the entropy of the distributions. (in classification this is a bit like looking for the most uniform predictive distributions) and prioritises those.
Entropy based active learning works ok but doesnt distinguish uncertainty that comes from a lack of knowledge (epistemic uncertainty) from noise. Techniques like Bayesian Active Learning by disagreement can do better. :)
Some examples that we've actually worked on/are working on: * Contract classification
* Content moderation
* NER
* Customer review understanding
* support ticket routing
Raza here, author of the post.
My high-level answer is weak-labelling overcomes cold starts and active learning helps with the last mile.
More detail:
We see weak learning as very complementary to active learning. By using labelling functions, you can quickly overcome the cold start problem and also better leverage external resources like knowledge bases.
But most of the work in training ML systems often comes in getting the last few percentage points of performance. Going from good to good enough. This is where active learning can really shine because it guides you as to what data you really need to move model performance.
At Humanloop, we've started with active learning tools but are also doing a lot of work on weak labelling.