Show HN: I made an NLP based essay grader
gradeflex.com
gradeflex.com
A+
Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like? Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like; Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like?
Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like.
Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like? Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like?
Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like; Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like.
Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like?
Grading is indeed time consuming, but it is one of the best opportunities that we have to give personalized feedback to our students. If we throw away that chance to analyze, consider, and connect, then we might as well substitute YouTube videos for all of education and be done with it.
Got me an A+, according to this algorithm.
EDIT: I'm not criticizing you! This is really cool, and you should keep working to improve it. Automating graduate students is definitely gonna happen, and this will bring us one step closer -- if we can work out the kinks. That said, it's always fun to poke and prod the AI nerds :D Makes them better!
For example, suppose we implement short range order detection, as you mention. Then, a very well written essay about the history of various metaphors used to describe General Relativity would receive a low score, because we'd be talking about trampolines, marbles, spider webs, and maybe even balloons at the same time. Your algorithm would immediately classify this as nonsense (similar to what I have written above).
The problem is not in detecting structural integrity. The problem is in detecting information integrity.
A nonsense essay about metaphors cannot be distinguished from a reasonable essay about metaphors unless your network knows what a metaphor is, how isomorphism works, and knows how various ideas relate to that.
Long story short, you can't hire an english teacher to grade a paper you're submitting for your GR course. Such a paper would be immune from random insertions of the word "not" or changing random real-valued numbers to other numbers, etc. Any NLP-only algorithm would immediately identify a false positive here.
On the other hand, a grammar-only-grader would be reasonable, because grammar is mostly independent of topic, meaning the AI would not need domain knowledge to grade it.
To understand language, AI is going to have be a lot more versatile than simply finding order within a paragraph, which is the most naive approach.
When a human reads a paragraph, they are playing with much more than simply the words contained inside the paragraph. IIRC, the ML community isn't there yet -- not because they don't know -- but because computing isn't there yet.
Got me an A-
But there is huge resistance to it because of Algorithm Aversion (http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort...). People really distrust statistical algorithms over (even really bad) human judgement. Unless they believe the algorithms are infallible.
One interesting thing you can do with automatic grading, is semi-supervised learning. If you know some students are good or bad already (perhaps from multiple choice, or from already graded essays), you can predict the grade of their essays. And then use the predictions to train a more accurate classifier.
Im not against algorithmic grading, per se, but I expect that we're at the phase (and will be for some time) where a human and a computer do a better job together than either does alone.
And I could imagine a consolidation of the grading tools, down to a small handful, or maybe even just one vendor. This is the pattern in textbook publishing. Once the human graders are laid off and their skill vanishes, education will be wholly dependent on the whims of those businesses.
Do you want to hire or work with 1000 engineers, who have all been through exactly the same training, and have adapted their styles to exactly the same template, worldwide? What if a new idea or problem comes along, that isn't addressed within the capabilities established by that template?
This is not too far fetched. I was in college when word processing and microcomputers were introduced to academia. I was one of the early adopters at my college. Both of these things radically changed how people did academic work, but a word processed essay might have gotten a low grade if the algorithms were tuned to the established preferences of teachers for reading hand-written essays.
https://media.giphy.com/media/3oz8xLd9DJq2l2VFtu/giphy.gif
It is fundamentally very different, as the former requires domain knowledge (i.e. to a larger extent) and the latter does not (i.e. to a lesser extent). It's just not that simple; even in doing the things you mention, the algorithm will be marginally better than a different algorithm.
The magic lies in the 1% that the algorithm misclassifies, not the 99% that it doesn't.
The ML community is centered around establishing high classification rates of static data sets. This will soon change, when people realize "99.9% accuracy on a fixed data set" is a meaningless metric for real artificial intelligence, not the monkey-see-monkey-do stuff that is currently considered "state-of-the-art".
There's about to be a big reckoning in this field, similarly to what happened in 1905.
Predicting the grade is one thing, assigning the grade is another. Remember the Sokal affair?
Any writing assignment that can meaningfully be scored by less than human-level AGI should not be assigned in the first place.
It seems like the criteria used here would favor longer, more complex, and more formal pieces going against most of the writing advice in Graham's piece above. These would generally match up with how writing is graded in schools though so it makes sense in that regard.
Incredibly cool project! Well done. I'd love to read more about the process of making this.
Disclaimer: I'm not associated with gradescope.
Turned out that none of the schools we contacted wanted to actually implement our system because 1: they didn't want to switch from all-paper to paper-digital now to all-digital in the future because education moves at geologic timescales, and 2: nobody had tried it before, so if something went wrong, whoever vouched for the program would get the blame.
Just something to consider if you plan to commercialize this application. CYA matters a lot on purchasing decisions.