Appreciation Combining Qualitative and Quantitative Lenses

Nicolas Rousselot and Alessandro Canossa - Massive Entertainment

Truth is in plurality of sources. Usability research can only be as good as the data you are using for it. It is not an opinion, not even an expert opinion that make a good usability evaluation, but the cold-reading of facts in order to extract meaning from players’ behaviours. But the interpretation of those facts might lead to introduce unexamined bias that can, unbeknownst to the researcher or analyst, lead to misinterpretation of seemingly obvious results.

Useless results are never pleasant, but misinterpreted results are downright dangerous, leading design teams’ astray and creating loss of time, efforts and confidence in User Research. Confronting and avoiding those issues should always be at the forefront of the Researcher’s or Analyst’s mind.

To that end, one of the most useful tools is the cross-referencing of several sets of data from different sources, so you can refine your analysis to be as sure as possible of your conclusion. For concrete examples, this talk explores various usability reports produced by the in the Massive Gameslab, and their approach to triangulation of qualitative and quantitative results.

This talk suggests exact methods that could, and should, be implemented in regular user testing, including examples of the advantages of these mixed-method approaches. The considered methods include analytics, direct observation, surveys and interviews and biometrics.

Being able to triangulate qualitative and quantitative results and adding more context to players’ issues by multiplying sources, while keeping it manageable, is quickly becoming a necessary skill for any Games User Researchers and Analysts in the industry. Nicolas and Alessandro will explore ways to combine these various methods of collecting data from players’ behaviours, and how they can work together. This will give teams direct and actionable (and more important, just) conclusions, to work on the design.

  • Category: Analytics / Business Models / Case Studies / Methods
Nicolas Rousselot, Massive Entertainment

Nicolas Rousselot is a Senior User Researcher for Massive Entertainment – a Ubisoft studio. Employed at Ubisoft since 2009, Nicolas worked first in the Editorial Gameslab from 2009 until 2014, where he tested most of the line-up of Ubisoft games during those years, from AAA title in the vein of Assassin’s Creed or FarCry 3, to smaller titles on 3DS or PC (Might & Magic games, Rabbids, etc.) as well as mobile gaming.

Since 2013, he started to work at Massive Entertainment where he helped create the Massive Gameslab, and organize, moderate and deliver results of the user tests for Tom Clancy’s The Division. While focused on TCTD, the Massive Gameslab kept testing various Ubisoft titles, as well as creating benchmarks, user studies, and methodologies for User Research.

His focus has been on usability, particularly qualitative testing integrated to games productions, as well as examining the accessibility of those games.

Connect with Nicolas on LinkedIn

Alessandro Canossa, Massive Entertainment

Alessandro Canossa is Senior Game User Researcher and Analyst at Massive Entertainment, a Ubisoft studio. In this role he coordinates the activities between the User Research Lab and the Analytics Group. He was previously Associate Professor at Northeastern University where he was director for the Master in Game Science and Design. He maintains a foot in academia as research professor within the College of Arts, Media and Design at Northeastern. He obtained his Master in science of communication from the University of Turin in 1999 and in 2009 he received his PhD from The Danish Design School and The Royal Danish Academy of Fine Arts, School of Architecture.

His doctoral research, carried out in collaboration with IO Interactive, a Square Enix studio, focused on user-centric design and evaluation methods. His research agenda can be broken down in three main areas: a) design and development of digital environments aimed at eliciting specific affective and/or behavioural responses; b) developing behavioural analysis methodologies that are able to account for granular social, spatial and temporal events, avoiding aggregation; c) design and development of visual analytics tools that can enable any stakeholder to produce advanced statistics and data-mining reports.

He published dozens of academic articles, garnering several Best Paper awards and he co-authored the book “Game Analytics” a milestone publication in a burgeoning field that is being used to evaluate user experience and behaviour in a variety of game environments including games for behaviour change, training, education and simulation.