When Two Heads Are Better Than One: How a CLS-funded Dataset Is Teaching AI to Help Us Think
We sat down with Professor Andreas Vlachos from the Department of Computer Science and Technology to talk about DeliData: a project that started with a gap in the psychology literature and has since reshaped how scientists think about group reasoning, deliberation-enhancing bots, and the future of AI.
Professor Vlachos was in conversation with Dr Chrysoula (Lina) Vassiliu, Postdoctoral Research Associate in the Department of Theoretical and Applied Linguistics.
What is DeliBot's origin story? Where did it begin?
It started with a book, actually. I was reading Daniel Kahneman's Thinking Fast and Slow, thinking about the failings of human reasoning and the cognitive biases that lead us astray. I mentioned this to Tom Stafford, a cognitive scientist at the University of Sheffield who became the other senior collaborator on this project. And he said: yes, but you know, there is another book by Hugo Mercier and Dan Sperber, The Enigma of Reason, that makes the argument human reasoning didn't evolve in isolation. It evolved in a social context.
That reframing was important. The psychological literature has long shown that groups of people, talking to each other, can outperform the same individuals working alone. So, I thought: someone must have recorded the conversations where this magic happens. We looked, and we found that no, actually, these conversations hadn't been captured, often because they took place in person, in psychology labs, long before digital communication made recording easy. That was the gap for us.
And that gap became the DeliData project?
Exactly. With seed funding from the Cambridge Language Sciences Incubator Fund, we set out to create the resource that didn't exist: the first publicly available corpus of small-group collaborative problem-solving conversations.
We used a task called the Wason Card Selection Task, a classic logic puzzle that has been studied in cognitive science for over half a century. It's a beautiful choice because it's well-established that most people get it wrong when working alone, due to confirmation bias. But when you put people in groups to discuss it, their performance improves substantially. So, it gave us a problem with a correct answer, a known individual failure rate, and a documented group benefit.
The protocol was straightforward. First, each participant solved the puzzle alone. Then the group chatted freely. Then each person submitted a revised individual answer. They weren't required to agree, just to reconsider. We incentivised genuine effort by offering a bonus payment for getting the right answer.
And what did you find?
The results confirmed the psychological literature, with the proportion of people who got the right answer tripling after deliberation. But what was really exciting was this: in 40% of the groups where at least one person ended up with the correct answer after discussion, nobody had the right answer before the discussion began. So, it was not just a matter of the person with the right answer convincing the others. By talking to each other, they were able to find better answers than any of them had found before. The group had collectively reasoned its way to something none of its members possessed individually. I think of that as a cause for optimism, actually.
The CLS funding, £3,500 in the first round followed by a second grant of £2,800 in 2022, paid for two things: compensating participants for their time, including the performance bonuses that made the incentive structure work, and hiring a trained linguist to annotate the resulting conversations. That annotation work was crucial. It built a schema to capture the specific linguistic moves, the probing questions, the moments of reconsideration, the expressions of uncertainty that seemed to drive deliberative improvement. By the time the dataset was published in the Proceedings of the ACM Conference on Computer-Supported Cooperative Work in 2023, it contained 500 group dialogues and over 14,000 utterances, all openly available online.
So, from the beginning, there was a longer-term goal beyond just building a dataset. What were you ultimately hoping to create?
Yes, the dataset was a means to an end. The real ambition was a Deliberation Enhancing Bot, a DeliBot, that could join a group conversation and improve its outcomes. Not by giving the answer, but by facilitating better deliberation. It would ask probing questions, encourage people to articulate their reasoning, surface disagreements that might otherwise go unspoken. More like a facilitator than an expert.
The point about not giving the answer is important. For a simple logic problem like the Wason Task, it wouldn't be an issue for any AI model today to just solve it. But the goal was to help people find the right answer themselves, not to hand it to them.
Georgi Karadzhov, a PhD student who led the data collection and annotation, built the first working DeliBot as part of his doctoral thesis. The results were encouraging: groups deliberating with the DeliBot showed a small but statistically significant improvement over groups working without it.
Where has the research moved since?
We felt that the Wason Task, while useful, is a very simple problem. We wanted something more complex, more engaging. So, we moved into chess. It has deep strategic complexity, well-defined rules, and objective measures of move quality via chess engines. But it's also a space of rich human disagreement, even among experts. That combination makes it ideal for studying how groups negotiate different perspectives.
We collected 107 group dialogues, with participants first solving chess puzzles individually, then discussing them together, then revising their answers. The findings echoed what we'd seen with the Wason Task: deliberation consistently improved group performance. And diversity mattered as well: groups with a wider spread of initial answers tended to benefit more from discussion. A DeliBot for chess is the next step.
How have your findings been applied to AI systems?
There is an increasingly popular technique in AI research called multi-agent debate, where instead of asking one language model to answer a question, you ask many of them, have them debate each other, and take the result. The intuition is the same as for human groups: surely multiple perspectives should do better than one.
In practice, it often doesn't work. And when one of my students, Xiaochen Zhu, came to me with this puzzle, I remembered something we had found in the DeliData analysis: that the diversity of answers among the human participants was strongly correlated with whether deliberation improved the group's result.
We checked whether the same held for language models. It did. When the AI agents all start with similar answers, debate simply reinforces the existing majority, the same conformity effect the psychological literature had long identified as a threat to good group reasoning. The fix was to deliberately seed the debate with a diverse initial pool of answers, and to have the models communicate their confidence, so that more certain responses carry more weight. Both interventions produced consistent improvements across multiple benchmarks.
This was a case where findings from how human deliberation works improved LLM deliberation. The traffic went in that direction, human insights improving AI, not just the other way around.
How important was the CLS funding to all of this?
The Incubator project provided us with the funding to collect the data with human participants. Without that, there would have been no corpus, no annotation, no DeliBot, and none of the work that followed. It also made the interdisciplinary collaboration possible. I'm not a psychologist, but working with Tom Stafford, who is, meant we could identify the gap and actually know how to fill it.
As the DeliData work was taking shape, we were also developing a larger follow-on project, which went on to receive £711,020 from the EPSRC. The two were growing in parallel, each feeding the other's thinking. The DeliData corpus also helped secure a $100,000 Google research grant; the dataset was exactly the kind of resource they wanted to support.
Finally, given everything happening with AI right now, how do you think about the stakes of this work?
I think we are reaching a point where most people have access to language models that know more than the average human on most topics. For a long time, some of us had the privilege of working with people more knowledgeable than ourselves, students, postdocs, colleagues, and we still had to make the decisions: where to submit a paper, what direction to take. Now that experience is becoming much more common.
The DeliBot project is, in some ways, preparing us for that relationship. Not AI that simply provides answers, and not humans who simply defer to them, but something more like the groups in the Wason Task experiments: diverse perspectives, honest uncertainty, and the kind of dialogue that turns a room full of wrong answers into a room full of right ones.
I know the discourse has doomers, but I'm on the optimistic side.
That optimism is shortly to find a wider audience. Vlachos and Stafford are putting the finishing touches on a book due from Cambridge University Press in December: Let's Talk about Facts: Trust, Collaboration, and Fact-Checking in the Era of Large Language Models. The CLS-funded work runs through it: the data, the findings, and the papers all contributed to the book and, Vlachos says, became a core part of the inspiration for writing it.