Show HN: Political Compass Quiz Powered by ChatGPT
yourpoliticalplace.com
yourpoliticalplace.com
It occurred to me while I was trying to explain myself to the multiple choice form that you could use ChatGPT, or similar LLM, for this. My original goal was "interactive questions" - so a question could ask "Do you think a wealth tax is a good idea" and the user could say "What kind of wealth tax, who would it apply to" and so on to refine the question to something they were comfortable answering with a yes/no.
Experimenting with this I discovered that it would actually take a lot of tokens (and therefore money) to have real conversations about the questions. Instead, I changed my vision from "interactive questions" to "short essay responses". Basically, you get a prompt for a political question, write your thoughts about that prompt, and then ChatGPT (3.5 - again, money) grades your essays on two dimensions according to a couple rubrics I wrote for it.
I have shown this to a few people and taken the test a couple times myself. I think it's working reasonably well. At the end of the test you can see how the LLM graded each of your essays and you can click "thumbs up" or "thumbs down" if you agree or disagree with the LLM. So far, I have 44 bits of feedback 5 of which are negative, which suggests to me that, for the most part, the LLM does a good job at grading.
One problem I have already noticed with the concept is that people often write answers that are much too short - e.g. some people will just respond by typing out "Yes", "No", "False" or similar one or two word replies. Such replies kind of defeat the purpose of the essay writing aspect and the LLM does have trouble fully grading them. I'm thinking about requiring a minimum number of words, or having the LLM prompt the user "Why do you think X?".
If I keep working on this one thing I'd like to do is replace the ChatGPT component with Llama 2. My understanding is that Llama 2 should be similar to or slightly better than GPT-3.5, so in my fantasy I can have a dedicated server running a possibly fine-tuned version of Llama 2 that will enable me to use a lot more tokens - grade on more dimensions, have more interactivity with the users, and so on. This probably won't happen though due to the cost of running such a server.
My other answers were more direct and the assumptions extracted from them were fairly accurate, so well done.
Perhaps there's a general difficulty for LLMs in recognizing the import of things that aren't said?
Without a good enough rubric 3.5 gives out basically random scores. I think if you are experiencing illogical scores it's likely pointing to a gap in the rubric. One way I hope to be able to improve is by looking at grades that are bad (users can click thumbs down if they don't like a grade) and using them to build a better rubric. I think GPT-3.5 is not quite smart enough to do the grading well. It needs a lot of hand holding. I was experimenting with GPT-4 versus 3.5 and 4 grades questions pretty accurately with no more instructions than
I want to test if its true that you really get a bit "more right wing as you grow older". I hope they keep it up for the next 20 years!