1,070 karma · joined July 12, 2019
On the other hand I agree that people who take the non-CS route may not be that intimately familiar with all the details.
On the topic of classification, large language models such as BERT are the state-of-the-art. But, there was a paper recently at the ACL [1] which showed that a simple approach of using gzip compression + k-nearest neighbour achieved similar level of performance. Later on, their KNN evaluation methodology came under question because instead of doing KNN with k=2 they reported accuracy based on top-2. Even then, what it shows is that we don't need LLMs for simple tasks such as classification, where traditional techniques still work well and are much cheaper to run. I'm sure some people might say that we should just discard that paper because it had that error. But Sebastian Raschka ran it on the IMDB movie reviews dataset and reported 71% accuracy [3]. I wrote an explainer article on it [2].
[1] https://aclanthology.org/2023.findings-acl.426.pdf [PDF]
[2] https://codeconfessions.substack.com/p/decoding-the-acl-pape...
[3] https://magazine.sebastianraschka.com/p/large-language-model...
For example if you want to do positive vs negative sentiment classification of movie reviews, this will perform just as well flipping a coin. (I tried this)
It works well for multiclsss classification. For example, let's say I have an article about soccer, I am a bound to have words like goals and scores. And I will find a good amount of texts in the corpus from the same class with same vocabulary. Thus their ncd will be lower, as opposed to texts from other classes. But this isn't really doing anything related to semantics.
I am going to be writing another article on this, where I will try to explain why gzip is helping it do this.
I mentioned it towards the end, in the 2nd last paragraph. Those issues in the evaluation do bring its accuracy down a bit, even then it performs better than expected, considering it is doing knn on compressed data.
Here are some of my posts which have been well received:
- https://codeconfessions.substack.com/p/creating-chatgpt-plug... : It takes you through a tutorial showing how to use the function call feature to build your own ChatGPT with plugins
- https://codeconfessions.substack.com/p/exploring-deepminds-a... - This explains the results of the AlphaDev paper from DeepMind
- https://codeconfessions.substack.com/p/mojo-the-future-of-ai... - This gives an introduction to the Mojo programming language
- https://codeconfessions.substack.com/p/will-ai-replace-progr... - This one is my personal take on whether programming jobs are in danger becaue of AI or not.
It helps if you have a back link to kickstart things. Even then if some of your internal pages are not easily reachable from the home page, those page will not get indexed soon enough.
As far as books are concerend: - Database Internals
- Designing Data Intensive Applications
- Disk-Based Algorithms for Big Data
- Database Systems by Ullman et al (http://infolab.stanford.edu/~ullman/pub/dscbtoc.txt) Part IV covers implementation details of a database system.
Thank you for sharing!
https://codeconfessions.substack.com/p/will-ai-replace-progr...