On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial

• https://arxiv.org/abs/2403.14380
• Francesco Salvi, Manoel Horta Ribeiro, Riccardo Gallotti, Robert West

Persuasiveness of generative AI models

In this pre-registered study, we analyze the effect of AI-driven persuasion in a controlled, harmless setting. … Each participant is randomly assigned to one of four treatment conditions, corresponding to a two-by-two factorial design: (1) Games are either played between two humans or between a human and an LLM; (2) Personalization might or might not be enabled, granting one of the two players access to basic sociodemographic information about their opponent.

We found that participants who debated GPT-4 with access to their personal information had 81.7% (p < 0.01; N=820 unique participants) higher odds of increased agreement with their opponents. Without personalization, GPT-4 still outperforms humans.