Plenary Panel: #gamesUR 10th Anniversary Discussion With Industry Pioneers
Speakers: Ramon Romero (SIE), Veronica Zammitto (Consultant), Jennifer Ash (Bungie), Mike Ambinder (Valve), Ian Livingston (Blizzard)
Session Time: 11:15-12pm EST (Day 1)
Ramon Romero started in Games Research in 1998 with Microsoft, working primarily on PC games and then orienting on the Xbox experience when that came along. As a Games Researcher Ramon has worked on titles as varied as Age of Empires to Kinect Adventures with some Munch’s Oddysee and Mass Effect thrown into the mix. In 2009 Ramon left Games Research and focused on the Xbox platform, shipping numerous updates to the Xbox 360 experience and then Xbox One and its first few experience revisions. Currently Ramon is an Experience Director with Sony Interactive Entertainment’s San Diego Studio. He manages UX Researcher, Experience Analytics, Strategy, UX Design and Game Designers, all working on the award-winning MLB The Show.
Veronica Zammitto is a thought leader in the user experience field, focusing on UX Strategy, organizational UX Maturity, and UX Research. She currently runs her own consultancy that specialized in helping companies to improve user experience strategies, scale UX efforts, enhance cross-functional collaborations, and elevate product quality. Veronica has been part of delivering successful products in video games (console, PC, and mobile), social media, and digital assistant tools.
Veronica is a former Director of UX Research at Electronic Arts, where she transformed how player experience is part of the development fabric by establishing novel methodologies, defining UXR processes, and scaling an international team across North America, Europe, and Australia. Veronica oversaw EA’s portfolio including titles like Battlefield, FIFA, Mass Effect, The Sims, Plants vs Zombies, Madden, NHL, NBA Live, among others, as well as the company's subscription services and websites.
Sharing knowledge and being inspired by others is very important to Veronica and that’s why she’s been presenting at numerous conferences including Game Developers Conference (GDC), IxDA, CHI, and has been part of every GUR Summit since it started in 2010.
Veronica is passionate about strategic UX improvements that lead to better company-wide processes and outcomes, delightful products, and building world-class UX teams.
Jennifer Ash is a Lead Designer at Bungie, working on Destiny 2. While at Bungie, she has held roles as User Researcher, UX Designer, UIUX Discipline Lead, Feature Lead and is currently Area Lead for World Systems. Outside of Bungie, Ash has contributed to multiple game projects, presented at a variety of conferences including the Games UX Summit and GA Conf, and taught at DigiPen Institute of Technology focused on user research and experience.
Mike Ambinder is a principal experimental psychologist at Valve who joined the company in 2008. He has a PhD in Experimental Psychology (with an emphasis on visual cognition) from the University of Illinois, and a BA in Computer Science and Psychology from Yale University. His work at Valve focuses on the application of knowledge and methodologies from psychology to game design, statistical analysis, and brain-computer interface research.
Ian Livingston is the lead user research team at Blizzard. He has mad skills in:
A Reseachers' Guide To Teaching Players (Without More Tutorials)
Speaker: Sebastian Long (Player Research)
Session Time: 1-1:45pm EST (Day 1 - Track 1)
This is a foundations talk aimed at early-career games user researchers. Players have to learn our game’s rules. It’s one of the few things that all our players have in common: they’re all going to play our game for the first time, and learn how to play. Sometimes this goes well. Sometimes not. As a games user researcher you’re going to be tasked with making sense of players’ behavior and cognition as it pertains to making your game learnable. You’re going to be the voice of players’ frustrations and misunderstandings, and thereby central in your team’s strategy on teaching and tutorialisation. Tutorials are powerful and reliable tools for helping players learn, but they have their limits. In this talk we are going to be thinking about the various ways players can be taught how to play. I’ll outline a very simple ‘mental model’ describing how players learn. I’ll explain how using this simple model as a ‘first principle’ will help you broaden your thinking on teaching players, improve your 'fixes'/suggestions, consider the limitations of research methods, and recognize the far-reaching implications of suggesting “we should just add this to the tutorial”.
About the Speaker
Sebastian Long is the Managing Director of Player Research. Seb splits his time between running Player Research's playtest labs in the UK & Canada, and consulting with gamedev clients of all shapes and sizes. He has run research studies for 200+ commercial titles, and designed research, training, workshops, and data strategies for a few hundred more.
Thinking Outside the Lab: A Case Study of a Mobile AR Game – Minecraft Earth
Speaker: Olga Zielinska (Xbox Research)
Session Time: 1-1:45pm EST (Day 1 - Track 2)
Minecraft Earth is a mobile AR game that involves walking outside in the real world to collect resources through tappables, battling mobs and mining ores in outdoor Adventures, and building a 3D Minecraft creation in your own home. All of these interaction methods are slightly different than the traditional console, PC, or even mobile games and require thinking outside the “lab”/player’s home environment when evaluating the player experience. During my presentation, I’ll talk through my experience testing an AR mobile game, the adjustments I made to traditional GUR methods to evaluate the player experience, and showcase our mobile lab set up that helped us capture gameplay on the go.
About the Speaker
Expanding Accessibility Through User Research in The Last Of Us Part II
Speaker: Kevin Keeker (Playstation Studios)
Session Time: 2-3:00pm EST (Day 1 - Track 2)
Building upon the foundations established with Uncharted 4: A Thief’s End, The Last of Us Part II features more than 60 accessibility settings, with expanded options focused on fine-motor and hearing, as well as completely new features that benefit low-vision and blind players.” The previous paragraph is from Naughty Dog’s website. In fact, the perspectives of many who were involved in the development of those accessibility settings are already available online in fascinating and informative videos. This talk’s unique focus is the approach taken by Naughty Dog and PlayStation Studios to apply user research to expand the accessibility of TLOU2. Collaboration on research started very early in production and continued throughout development. Developers with a wide set of roles and responsibilities were engaged in the research process. A broad, inclusive set of participants contributed feedback. User research methods were applied to understand accessibility needs, iterate strategies, and validate solutions.
About the Speaker
Kevin Keeker specializes in bespoke game-improvement research, efficient real-time research methods and accessibility research. He has presented at the Games Accessibility Conference and organized and participated in sessions for HCI-I Games, GUR Summit and CHI. Before PlayStation Studios Kevin worked at Zynga for 3 years and Microsoft for 16 years. He has had brief, eye-opening stints as a game designer. Kevin’s academic training is in Social Psychology with an emphasis on statistical analysis of mixed-method behavioral measurements. Recent products include: The Last of Us Part II, Days Gone, God of War, Horizon Zero Dawn and Uncharted 4: A Thief’s End.
Hold onto your butts! A rapid fire debate on the virtues and pitfalls of teaching teams to do their own research
Speakers: Ian Livingston (Blizzard Entertainment), Tom Lorusso (Xbox)
Session Time: 3:15-4:15pm EST (Day 1 - Track 1)
The discipline of user research is constantly evolving. But what’s the right direction of that evolution? Do we hold on to our craft for dear life, or share it with the rest of our product teams? Do we double down on growing the number of researchers in a company, or empower those who consume insights to contribute in their creation?
The democratization of user research is a hot topic in the games industry and who better to debate it than two senior leaders with opposite viewpoints. In this debate style talk, Ian Livingston from Blizzard and Tom Lorusso from Xbox will present the points and counterpoints to a central thesis:
The future of user research in gaming is to empower everyone to conduct research and create people centered insights.
The goal of this talk is to explore both arguments and give the audience two different views of a complex and future facing topic. It should also feel a bit like a back room discussion, since that’s how it started: with Tom and Ian discussing the future of the industry, and finding only one topic they really disagreed on.
The presentation will have a debate format – there will be opening statements and rebuttals from both sides, and then a Q&A section between the two researchers.
About the Speakers
Ian Livingston is currently the Sr. Manager for the User Research Team at Blizzard Entertainment. He and his team of researchers support multiple franchises including World of Warcraft, Overwatch, and Hearthstone. Previous Ian lead the Canadian UX research team at EA, where he partnered with studios and team such as FIFA, Motive, and BioWare. Prior to EA, Ian worked at Ubisoft Montreal as the Research Lead for the Far Cry franchise, working on titles including Far Cry 3 and 4. Ian's expertise and experience is board; ranging from user experience research & design, human computer interaction, and the effects of Game media on player perceptions, to the design and development of gameplay telemetry systems, and the integration of research production & business processes. Ian has been working in the video game industry for almost 10 years, has published numerous academic articles in the field of HCI, and has presented work at a variety of conferences including GDC and SIGGRAPH.
Tom Lorusso, Xbox Tom Lorusso is a Principal User Research Manager in Xbox at Microsoft. He and his team are focused on games and gaming experiences across the entire Xbox ecosystem and Windows. Tom has been working in games for over 9 years, with 20 years of total experience in the User research field. He has been a researcher on phones, operating systems and portable ultrasound. Tom and the User Research team for Xbox Experiences tackle a variety of research questions for areas including social gaming, community, monetization, communication, cross platform gaming, video consumption, accessibility and inclusivity and of course, designing engaging games. He is also a frequent presenter at conferences and universities and is always happy to sit down and talk about games research with anyone who wants to listen. Tom is currently playing way too much Fortnite and Hearthstone and is using any excuse to jump on Xbox to talk to people who don’t live in his house.
Forest Fires: A Usability Post-Mortem of Root
Speaker: Joshua Yearsley (Leder Games)
Session Time: 3:15-4:15pm EST (Day 1 - Track 2)
Root is an acclaimed asymmetric strategy board game—so let's talk about all its mistakes! Joshua Yearsley, Root's usability developer, will tell some stories of Root's development gone right and wrong, and what game developers can learn from his mistakes in rules writing, graphic and component design, user experience research, and more. He will raise questions about how user research practices in board games can inform those in video games, especially as video games and board games continue to draw inspiration from each other, and about how we can foster healthy communities when creating games that draw from both mixes of hardcore and mainstream audiences. This talk is suitable for both newer practitioners (since it contains plenty of rookie mistakes!) and veterans alike, especially those with limited exposure to tabletop game UX work.
About the Speaker
Joshua Yearsley (Leder Games). Josh is a developer and editor who has worked in tabletop games for eight years. He has contributed to over a hundred games, including the Leder Games hits Root, Vast, Fort, and Oath, as well as many others from top studios including Asmodee, Fantasy Flight Games, and IDW. Before becoming a developer, he was a published scientist. Bringing together his scientific background and love for games, he is equally at home working on articles about modulating Fermi level pinning at metal–oxide interfaces and on games about noir investigations and druidic magick.
Why doing the boring stuff you think is not your job, really is your job
Speaker: Emma Varjo (Frozenbyte)
Session Time: 4:30-5:15pm EST (Day 1 - Track 1)
When we think of our jobs, or possible future jobs, we think of a subset of tasks and things we actually do. Things that are easily and clearly defined, such as "creating prototypes" or "planning research". But what about other stuff? The meetings, both formal and informal, we need to attend, documentation and reports we need to write etc. Those aren't usually the first things that come to mind when we think about work, though those are arguably very important. Those are also often not something you get explicitly taught. This talk focuses on just this kind of "hidden" work, and how to become better at it."
About the Speaker
Emma Varjo is the UX Lead at Frozenbyte. She works hard to ensure great UX of the games made at Frozenbyte by using her knowledge of Human-Computer Interaction, running playtests and gathering feedback from players, and generally just taking to people a lot."
Game Playing Preferences: From Player Types to Player Traits
Speaker: Maria Karina Arrambide Figueroa (Ph.D. Student, University of Waterloo), Lennart Nacke (Director of HCI Games Group, University of Waterloo)
Session Time: 4:30-5:15pm EST (Day 1 - Track 2)
The Games User Research (GUR) community has been collectively studying and classifying player preferences to understand what playing styles and game elements are enjoyed by what people. This knowledge can help designers create games better targeted to their audience, so they can offer their players the content they want; marketers segment their player base, so their campaigns can be more effective; and researchers explain the variables that influence the player's experience and enjoyment. This can also lead to the design of more effective games with a purpose, such as educational or health-related games. But despite the efforts of the community and the recent advances, we still lacked a player preferences model that is backed by empirical evidence and a validated measurement instrument. Given these shortcomings of the existing literature, we created and validated a player traits model with an accompanying measurement scale. Our Five-Factor Player Traits model describes the traits that explain what kind of experiences each person prefers when playing games: aesthetic orientation, narrative orientation, goal orientation, social orientation, and challenge orientation. Additionally, we created a 25-item survey that can easily be utilized to determine the trait scores for an individual. This is the first model based on player traits instead of types, which better captures the full range of individual preferences because it recognizes that individual preferences are a combination of multiple traits, rather than a single type. This talk describes the five player traits and how they can be used in game design, marketing, and research.
About the Speakers
Maria Karina Arrambide Figueroa is a Ph.D. student pursuing a degree in Systems Design Engineering at the University of Waterloo, under the supervision of Dr. Lennart Nacke. She holds an MSc in Information Technology with Business and Management from the University of Sussex in the UK, and a BSc in Information Technology from the University of Monterrey in Mexico. Her main interests include understanding player's behaviors and emotions by applying diverse games user research methodologies, specifically biometrics such as electromyography and galvanic skin response. She is also interested in the research of new methodologies and technologies that can help improve player's experience.
Professor Nacke teaches User Experience, Human-Computer Interaction, and Game Design at the University of Waterloo. As part of the Stratford School of Interaction Design and Business, the Department of Communication Arts, and the Games Institute, he is researching player experience in video games, immersive VR environments, and gameful applications. As a truly interdisciplinary researcher, he is cross-appointed and supervises graduate students in the Department of Systems Design Engineering, the Department of English Language and Literature, and the Cheriton School of Computer Science. Together with co-researchers, he published the PXI — player experience inventory, gamification user types hexad scale, guidelines for biofeedback and sound design in games, and a book on games user research. Professor Nacke has served on the steering committee of the International Game Developers Association Games Research & User Experience Special Interest Group in the past, and was the chair of the CHI PLAY conference steering committee from 2014–2018. His publications have won Best Paper Awards at the CHI, CSCW, and CHI PLAY conferences. He has published more than 100 scientific papers, which have been cited more than 10,000 times. He strongly believes in understanding users first to build more engaging games and compelling player experiences.
Usability and Accessibility Pro-Tips for Games User Research Lab Design
Speaker: Ashley Brown (PhD), Entertainment Arts and Engineering/ University of Utah
Session Time: 5:30-6:15pm EST (Day 1 - Track 2)
Ever wonder how you can make your user research lab more accessible? Well you’ve come to the right talk. After listening to this half hour presentation of survey results you will walk away with a list of 7 pro-tips on designing a user research lab that’s usable, accessible, and accommodating to playtesters of all shapes, sizes and abilities.
The data presented here comes from a survey conducted in April 2019 which asked students, faculty, staff, and members of the Games User Research SIG how to create the ideal playtest lab. The survey consisted of 10 mandatory multiple choice questions which involved selecting images of desirable furniture, drapery and paint colors and 9 short answer optional questions which asked for selection justification/explanation. Although the furniture choices were interesting, the short answer form was an absolute gold mine of accessibility and usability tips to ensure every lab can be used by everybody. The results of this survey were ultimately used in making furniture, paint, and carpet purchasing decisions for the Games User Research lab at the University of Utah, but I also wanted to give the knowledge I gained back to the GUR SIG community with this presentation. If you ever are put in the position of needing to buy furniture or remodel a lab space, this presentation is a great reference point and reminder to keep accessibility and usability concerns at the heart of your purchasing decisions.
About the Speaker
Dr. Ashley Brown is an Associate Professor (lecturing) at the University of Utah where she teaches in the Entertainment Arts and Engineering program and runs a Games User Research Lab which mostly tests student games. When she isn't teaching, researching Twitch, or volunteering her time to make games for underrepresented communities, she enjoys walking her dogs.
Players are not lab rats! A retrospection of models concerning user experience
Speaker: Laureline Chiapello (PhD), UQAC - Université du Québec à Chicoutimi
Session Time: 11:15-12:00pm EST (Day 2 - Track 2)
How do you define experience in “user experience”? In recent years, controversies about gambling helped raise awareness concerning video game user experience and put forward disciplines such as psychology and design.
However, when we look at which scientific theories are used to understand UX in games in the industry, the diversity and progress made in academia are not reflected. The vision of user experience tends to be oversimplified, sometimes using outdated philosophy. Most companies still mainly use the behaviorist “stimulus-response” model when envisioning their user experience. Using the case of the user’s experience of loot boxes, the talk will demonstrate how experience is almost invariably considered from a behaviorist point of view, using Skinner’s research on operant conditioning.
This presentation will show the limits of the behaviorist model, explain the evolution of theories, and suggest an alternative: the pragmatic model.
About the Speaker
Laureline Chiapello (PhD) is Professor of Game Design, Playability and Narrativity at the School of Digital Arts, Animation and Design (NAD-UQAC), Canada. She obtained her PhD in design from the University of Montreal, Canada. She is also a member of PRAXIS Lab, and regularly collaborates with video game studios. Her research focuses on the various practices of video game designers – their creative processes, knowledge, roles and training. She employs a pragmatist approach to design and creativity, in a transdisciplinary fashion. She is particularly interested in the philosophical underpinnings of game design theories.
Climbing mountains - 10 years in accessibility
Speakers: Steve Saylor, Ian Hamilton
Session Time: 1-2:00pm EST (Day 2 - Track 1)
Accessibility is a rapidly evolving field. The pace of change is exponential, across the years that the summit has been running for it has progressed and changed beyond all recognition. This talk will share a greatest hits collection of some of the key moments in accessibility over the past ten years, to give a sense of that change - where the field has come from, where it is today, and what kind of trajectory it is on for the future.
About the Speakers
Steve Saylor is an accessibility advocate, consultant, content creator and Twitch Partner. He has consulted with studios such as Ubisoft, EA and most recently Naughty Dog on “The Last Of Us Part 2.” His YouTube series started in 2015, “BLIND GAMER” has educated viewers in how Steve is able to play games as a visually impaired gamer. As well as educating viewers on the importance and knowledge of accessibility in video games. He’s been featured on CNN, CBC, BBC, NPR, USA Today, VICE, IGN and Gamespot. You can watch him live on Twitch twitch.tv/BlindGamerSteve, find him online @stevesaylor on Twitter, and his YouTube channel youtube.com/snowball.
Ian Hamilton is a game accessibility specialist with a 15 year background in raising the bar for gamers with disabilities, though advocacy and awareness raising – writing, speaking, organising events, community building – and consulting, working with studios from the smallest indies to the largest AAAs, with publishers, platforms, industry and government bodies. Co-director of GAconf and coordinator of gameaccessibilityguidelines.com.
Terraforming the collaborative UX research landscape: Academic-Industry partnerships for positive change
Speakers: Pejman Mirza-Babaei (UXR Lab), Samantha Stahlke (Ontario Tech University)
Session Time: 1-2:00pm EST (Day 2 - Track 2)
Two years ago, I proposed a GUR Summit talk on negotiating academic-industry collaborations, focusing on IP, publicity, data-sharing, and other contract complexities. In the time since that proposal, the world, and the research landscape with it, have changed drastically. While it is obvious that research collaborations are still instrumental in our field, I felt that I could better use this opportunity to explore how these collaborations can fuel positive social change and empower game researchers moving forward in this new world. A global pandemic has brought people all over the world to an online workplace out of necessity. For games user researchers, this means a scramble to figure out remote testing workflows, but it also presents an incentive for more global collaboration—with teams all over the world driven online, working with someone on the other side of the planet might be just as easy as calling up a neighbour. Amidst these sweeping changes to how we communicate, we’ve also witnessed the rise of movements like Black Lives Matter to demand justice in the treatment of marginalized communities. Fittingly, we are seeing a long-overdue urgency to act on related issues in the games industry, such as addressing systemic issues in the workforce, representation in the products we create, and the attention given to accessibility concerns. These are significant challenges requiring significant effort to address, beyond what any one organization could hope to achieve. Thus, this talk will focus on how we can work together through collaboration across institutional borders to create a more connected, inclusive, and productive community in the industry and in games research as we enter the next decade.
This presentation is co-created by Samantha Stahlke and myself, with contributions on the narrative from Anders Drachen, Neha Kumar, and Lennart Nacke.
About the Speakers
Pejman Mirza-Babaei is an Associate Professor of UX research and Associate Dean for industry partnerships at Ontario Tech University. His research and professional work are focused on evaluating user experience in games, and have been carried out in collaboration with companies spanning the industry. He has worked on the evaluation of more than 25 commercial games, and is a co-editor of The Games User Research Book. He has worked as the UX Research Director at Execution Labs (Montréal, Canada) from 2015-17 and UX Researcher at Vertical Slice/Player Research (Brighton, UK) from 2009-13.
Samantha Stahlke is a research assistant at Ontario Tech University in Ontario, Canada. She completed her Master’s research in the development of tools for automated playtesting with AI models of player behaviour. In addition to research, Samantha has worked as an indie developer and university educator. Her primary research interests include applications of AI in GUR, evolving GUR methods, and exploring design for emerging platforms.
Borderlands 3 User Research: Revisiting old methods and innovating new approaches in franchise testing
Speaker: Jonathan A. Cohen (Gearbox Publishing)
Session Time: 2:15-3:15pm EST (Day 2 - Track 1)
Gearbox User Research tested Borderlands 3 using new methods in combination with best practices from over ten years of franchise research. Using a series of case-studies we illustrate how established and new approaches can generate insights that inform development.
Borderlands 3 added mantling and sliding to character movement. Traditional playtests helped reveal that these mechanics were not well-understood; developers added a mission that reinforced tutorials.
Similarly, Borderlands 3 introduced changes in how the player-character’s Action Skill functioned; we observed that players had difficulty with the new system. Playtesting contributed to the iteration of tutorials that improved comprehension.
A case-study that highlighted the importance of teaching even veteran franchise players mechanics involved bird creatures in a boss fight. Players were unaware that these creatures could be attacked to gain a “Second Wind.” Developers crafted a moment where an ally shoots them to demonstrate, after which players began to imitate the action.
We innovated our approaches to answer an audio question regarding whether players could perceive the differences among gun sounds. Taking a page from signal detection work we conducted a study and learned that these sounds were distinguishable from one another in the absence of visuals.
Borrowing from navigation research we conducted a study in the game’s hub map. Participants learned the layout of the hub by visiting locations, then completed tasks to reveal whether the layout led to confusion or errors.
Improvements to long-form testing had positive impacts on narrative. Player interviews and discussions revealed that the primary antagonists were less prominent in the middle of the game. Changes derived from this feedback led to improvements in player engagement and narrative comprehension.
Viewers should take away from this talk recommendations for how to integrate new approaches with existing methods to better inform development.
About the Speaker
Jonathan A. Cohen is the User Research Manager at Gearbox Publishing; he has a background in cognitive science and experimental psychology with over 19 years of laboratory research experience that he applies to games user testing at the studio. Outside of games research he enjoys movies, craft beer, tabletop RPGs and board games, and he looks forward to one day returning to in-person gaming with friends and colleagues.
Group Therapy for Old-Timers
Speakers: Jonathan Dankoff
Session Time: 2:15-3:15pm EST (Day 2 - Track 2)
Join several GUR veterans for a frank discussion about the challenges they face as they approach the "Top of the Ladder". The format will use a live polling tool to allow many panelists to share their opinions, and then dive deeper on the most thought-provoking concepts. This talk is meant for seniors of the discipline and will cover broad topics such as "what comes after research director?", "how do we evolve the discipline?", "what motivates you?", and "why do my knees always hurt?".
About the Speaker
Jonathan Dankoff is a games user researcher with 15 years experience, including a quick decade at Ubisoft, working primarily on the Assassin's Creed franchise, a stint managing the wonderful research team at WBIE, and then a brief visit to Stadia Games & Entertainment. He's currently working on something secret but is very excited about it.
Analytics and Experimentation for Games UR
Speakers: Andrei Muratov (WB Games)
Session Time: 3:30-4:30pm EST (Day 2 - Track 1)
Analytics and User Research share many common goals -- understanding player behavior, motivation and providing dev teams information with which to improve player experience. At WB, UR and Analytics collaborate during studies to cover the observable and unobservable behaviors of the player and the game systems. In many cases, this helps us observe the unobservable -- behavior of game systems, AI, and other unseen modifiers of the player’s experience. With a robust analytics system and a modern, connected title, we can leverage the game systems to experiment with players to find best tunings and versions of the game simultaneously. Compared to a test-analyze-adjust-test cycle, experimentation performs just one cycle and provides more conclusive and actionable results sooner.
About the Speaker
Andrei Muratov leads the Game Analytics group for WB Games, supporting dev teams with insights on player and game system behavior during development and live ops of premium and free-to-play games. Before getting into analytics and experimentation, he worked as an artist, producer and designer at a variety of startups and indie studios.