Comment by bane | Hacker News Reader

Comment by bane | Hacker News Reader

One of my first jobs was helping build an expert system for a, even today, complex computational linguistics problem. The company had a rich corporate library full of academic books on expert systems, decision trees, first gen (pre-winter) AI, and some early books on early ML approaches. I remember seeing this book in particular and its evocative title caused me to look deeper into the library than I would have normally.

Our core system was built of thousands upon thousands of hand-crafted rules informed by careful statistical analysis of hundreds of millions of entries in a bulk data system.

Part of my job was to build the system that analyzed the bulk data and produced the stats, and the other part was carefully testing and fixing the rulesets for certain languages. It was mind-numbing work, and looking back we were freakishly close to all the bit and pieces needed for then bleeding-edge ML had we chosen to go that way.

However, we chose expert systems because it gave us tremendous insight into what was happening, and the opportunity to debug and test things at an incredibly granular scale. It was fully possible to say "the system has this behavior because of xyz" and it was fully possible to tune the system at individual character levels of finesse.

Had we wanted to dive into ML, we could have used this foundation as a bootstrap into building a massive training set. But the founders biased towards expert systems and I think, at the time, it was the right choice.

The technology was acquired, and I wonder if the current custodians use it for those obvious next-step purposes.

Your post is making me think maybe there is quite a lot of lost knowledge out there somewhere that maybe has pertinence in modern day agentic AI system building use. I am currently experimenting in building my own AI system that uses LLM's as the "engine" but the "harness" around the said LLM will do most of the heavy lifting. It will have internal verification systems, grounding information , metadata, etc... And I find myself making a lot of automated scripts as part of that process as I have a personal motto that its always better to automate everything possible with scripts first and only use LLM's as a last resort or for things you can script away. And that is making me look more and more in to old techniques that have been long established way back when...

Well from what I remember from university most of expert systems went into bust because they were promising what ML today promise.

The maintenance of the rules or for you scripts for complex tasks is much more work than anyone is willing to commit to. Also big problem was finding out tacit knowledge and no one was able to code that reliably in.

ML today is promising you won’t have to hand code the rules you just push data and system finds out what the rules are and then can handle new data.

I don’t have to code the rules to check if there is a cat in the picture - that definitely works. Making rules on data that is not so often found on the internet that’s still going to be a hassle. Rules change and world change and for example knowledge cut off is I think still a problem.

In the end yes you can build nice system for some use case where you plugin LLM for classification and you most likely will make money on it. This just won’t be „what was promised” so AGI and we are stuck with this promise and a lot of people won’t accept less than that.

I wonder what would happen if you used an LLM to write the rules?

That's exactly what I am focusing on. IMO getting rid of the biggest pain points regarding script building but also benefitting from modern day AI systems. "Have your cake and eat it too." We know that scripts are far more reliable then LLM's, but to build a good complex script is a pain in the ass and takes tremendous effort in creating and maintaining/debugging. So we use leverage modern day generative systems and have them be the builders and maintainers of said scripts. So the most intensive part for a human now comes down to creating a robust AI system that is able to build scripts reliable and then use them in conjunction with its own generative capabilities. Basically teach the machine to create the tools it will use and maintain, after that drop the human from the loop.

Your post is making me think maybe there is quite a lot of lost knowledge out there somewhere that maybe has pertinence in modern day agentic AI system building use.

I agree with that. In fact, that mindset is what led me to this book in the first place. I was exploring an older book on OPS5[1] and saw this book mentioned, and started looking for it and found that it is freely available online. Seemed like something the HN crowd might enjoy, so here we are.

And that is making me look more and more in to old techniques that have been long established way back when...

I suspect that there is some meat on that bone. I'm exploring this particular area as well. I think there's some opportunity for hybridization between LLM's / GenAI and some of these older approaches.

[1]: https://en.wikipedia.org/wiki/OPS5

I spent several years working with OPS5 in the 1980s. The Common Lisp code, especially the Rete network stuff, was fairly straight forward to modify and generally work with. Good times.

Back when expert systems were relevant, we had nowhere near the amount of compute that's basically required for modern day ML. Even if the data somehow was in place for it.

Reply on news.ycombinator.com