Case Study - A / B Testing of AI Communication Styles
I studied AI communication styles in Rocket League, an online, team-based, sports game.
- Client
- Clemson University
- Industry
- Technology
- Year
- Artificial intelligence (AI)
- Statistical analysis
- Project management
Overview
- Rocket League is an online, team-based, sports game
- Led three research projects on AI communication—this project focuses on AI’s communication modality
- Stakeholders included a Research Supervisor, a Human-Centered Researcher, and a Software Engineer
Rui produced novel insights on AI communication in human-AI collaboration by examining AI’s modality, proactivity, and explainability.
Director of TRACE Research Group
- Projects
- 3
- Teams
- 150
Research questions
- How do people collaborate with AI using text vs. visual communication?
- How were team coordination and performance affected?
Research process
- Proposed research questions based on existing work in the field and refined them with stakeholders given the implementation limitations
- Developed the research plan and collaborated with the software engineer to implement the AI agent
- Conducted pilot tests for AI agent QA, refined the project, and collected data with 100 participants
- Performed statistical analysis using mixed-effect models in R Studio
- Published the results in the prestigious CSCW journal
Key metrics
- Demographics: attitudes towards AI
- Teammate metrics: trust, communication quality, performance
- Team metrics: workload, performance, viability
Many other metrics were also reported…
General Outcomes
- AI visual communication was rated lower than AI text communication
- However, AI visual communication contributed to higher team performance
- Users feel more disconnected when collaborating solely with AI teammates
Gender differences
- Women perceive their human teammates to have better individual performance than men
- Women tend to perceive the task to have a higher workload than men
- Men performed better than women with a human and an AI teammate