Case Study - Usability Testing of an AI Learning Partner

Overview

• Jigsaw Interactive Agent (JIA) provides realtime support for small groups of middle and high school students in collaborative work
  • Led the user experience research throughout the entire product design and development process
• Evaluated the usability of JIA to improve the UI and functionality
• Stakeholders included the Project Manager, a UX Designer, Natural Language Processing (NLP) and Multimodal AI Experts, Software Engineers, Learning Scientists, and a Recruitment Manager

Research questions

• How do users perceive the design and functionality of the AI partner JIA?
  • How can we improve the trustworthiness of JIA?
• What are users’ reactions to JIA's support?
  • What issues do users encounter when working collaboratively with JIA?

Research process

Managed two phases of usability testing:

• Wizard of Oz to test AI partner JIA's interface and interaction design
• Human-in-the-loop to evaluate JIA's collaborative functionality

Usability testing procedure

• Group introductions and ice breakers
• Individual learning work
• Collaborative learning work with support from JIA
• Post-survey feedback
• Focus group feedback

Key metrics

• Reaction to JIA
• JIA’s trustworthiness
• Collaborative task performance
• Time spent on task

Many other metrics were also reported…

Positive feedback

• Users indicated the AI partner JIA’s collaboration was helpful and supportive
  • JIA supporting users by asking relevant questions was perceived positively
• Users perceived JIA to be trustworthy (4.2/5)
• Users indicated that the JIA web app was simple and easy to use

Actionable feedback

• 20% of JIA's message audio notifications were missed during active and rapid conversations
• Ask for help button should accept detailed user input, rather than be a toggleable button
• Personalization of the AI Partner