camrobjones

turingtest.live is back up!

The new site uses a 3-party format where you chat with both a human and an LLM at the same time, and your task is to decide which is which. This setup is closer to Turing's original idea, and we think it will be much harder for the models to pass.

We've also added a range of models, including GPT-4o, Claude, and LLaMA, along with new prompting techniques that allow the models to interact in different ways.

The site will be live every day from 12–1 PM and 8–9 PM GMT (8 AM and 3 PM ET, 5 AM and 12 PM PT). We've done this in the hope of increasing the density of users online at the same time.

I actually agree that Turing's 1950 definition is pretty vague in places and there are a few different interpretations out there. As we discuss in the paper, it's also unclear what should constitute a pass in terms statistical analysis.

I think the key thing to remember with the human benchmark is that it will change depending on interrogators' assumptions about human abilities.

When models are bad, humans are easy to spot. But if the model's pretty good, it's harder to be sure you're talking to a human.

On top of that, I think a lot of human users didn't want to get got by the model, so they had an a priori bias toward saying AI.

This is a good point. We didn't have any cases of GPT-4 failing to try to pretend to be a human, although GPT-3.5 did this quite frequently.

I think RLHF causes other issues though (especially with the tone), and we're interested to try some non-RLHF'd open source models.

Thanks for pointing this out, we do discuss it briefly in section 4.2, but I like your analysis in the blog!

Since the models are not passing, we didn't think it was a huge issue. If models are consistently passing the 2-person version, I think the motivation for running a (more cumbersome) 3-person version would be a lot stronger.

Results from a public online Turing Test show GPT-4 passes 41% of the time, falling short of chance and the human baseline (63%).

Thanks!

Thanks for the feedback and that's a great insight! The models will sometimes say they don't know but are probably much more likely to engage on obscure topics than a typical human.

Really excited to share turingtest.live, a site where you can play the Turing Test with GPT-4. You get randomly matched with either a human or an AI, and you have 5 minutes to decide which it is.

The platform is part of a research project I am carrying out on LLMs ability to interact like humans. While Turing originally developed the test as a way to measure machine intelligence, we're also interested in the extent to which models can deceive humans, and effective strategies for spotting and preventing deception.