Show HN: Using prediction margins to decide when to trust your classifier
jeffyclassify.com
jeffyclassify.com
I spent a lot of time in the 2010s trying to revive symbolic A.I. I rejected (and still reject) the usual conclusion that knowledge base construction is too expensive, but I found the lack of a rigorous and systematic approach to dealing with uncertainty was fundamental... And boy did I learn that half-baked approaches to the problem didn't quite work!
> I rejected (and still reject) the usual conclusion that knowledge base construction is too expensive
This is very interesting. Even though it's not the same, I feel MS' Phi-1 is a bit of demonstration of the impact that high quality data has on training
The cool thing vs 2010 though, is that there are so many resources and you can do so much stuff locally. I keep my M3's MPS and CPU busy most of the day and night doing training runs with models in the 800k-40M params size. And lately I've been offloading to cloud GPU too (incredible that you can get an H100 for about $2/hr). So it's a great time for individual/local research
Would love to know more about what you've worked on before? Care to share any links?
https://www.slideshare.net/paulahoule
note there are some good ones on page 2. The big picture was that my biz dev guy and I believed it was possible to build a system with a feedback loop to build symbolic AI databases and that this could be applied to data integration, low/node-code and a few other business areas.
In the midst of this whenever money got tight I would up working on neural nets for people!
I definitely recognized you could use the same feedback loops to train and eval machine learning models and in fact the idea was these were going to integrate with the symbolic models. In 2017 I worked for a startup that was creating very similar feedback loops around CNN-based neural network models.
What I did end up producing in that time is an ISO document describing how to use semantic web modelling for describing document formats which could have been a replacement for the system they were using for ISO 20022 -- so it felt like a second PhD when it was all said and done. I am definitely thinking of writing up a retrospective of the time because the vision makes more sense to me than it did back then and was certainly ahead of it's time... Like in 2019 I thought that "Chatbots in 2017" talk aged poorly but once ChatGPT came out it was... prophetic!