Test Case Scenario: Dr. Deborah Hendersen

A user researcher at Microsoft breaks down how game testing affects design.


Credit: Dr. Deborah HendersonCredit: Dr. Deborah HendersonThere’s often a deep divide between a game designer’s goal and a player’s expectations, and that’s where behavioral researcher Dr. Deborah Hendersen comes in. Hendersen is charged with the not-so-simple task of figuring out if users are connecting with Microsoft’s broad array of games and if so, how and whether they’re having a gameplay experience that matches with the designer’s intentions. Using a mix of science, observation, and deductive reasoning, Hendersen and the user researchers at Microsoft help the studio find ways to help players grasp game concepts more intuitively and work to distill down what makes a game fun…or not. Over the past few years, Microsoft has changed the way that it researches character and narrative development. What Dr. Hendersen has found may surprise you, and help you build a better, more memorable game.

Get In Media: Walk me through the process of doing behavioral research on a game. What are you looking at in terms of the developer end and what are you looking at from the user end?

Dr. Deborah Hendersen: The first thing that happens, obviously, is a conversation with design. We ask them, what are you trying to do with this? How we test things depends on the questions that the designer has. Really early on, we generally do tests that are focused around something we refer to as the core mechanics of the game. If it’s a racing game, can you drive the car? Can you figure out how to accelerate? Can you break? If you’re in a fighting game, can you actually hit things with your sword? Do you understand when you’re getting hit? These questions are not about is the game fun or is the experience fun, but rather about just are they functional? Can they work? Do they make sense? Do they seem to promise things that the game isn’t actually trying to promise or imply? The first question is, really, can we figure out whether the game is communicating with players in a way that makes sense to them and also in a way that’s very functional for them?

The next thing we like to test are the learning systems. When I’m testing for mechanics, I’ll do this on a one-on-one situation where I have a player come in and I have them do a bunch of tasks with whatever build we have in whatever state it’s in. What’s great about that is we can get a lot about understanding because we ask people to think aloud and we can really understand what the player thinks is going on and identify any sort of mismatches between the way design wants it to be working and the way it’s actually working. However, in order to get at questions around emotional things like is a game fun? Is it exciting? Is it fast in terms of its pace? Whatever those goals are for the designer, we need to be able to test lots of people at once because the data we need to collect differs.

Once we get that worked out, we start looking at what the experience looks like. We start off with a small snippet of it, generally the initial experience because that first hour is really important. It sets the tone for a game. Then we move into extended experience testing where we’ll test the entire game. We’ll bring people in for multiple days and have them play through as much of the game as they can possibly get through, breaking at various points and asking them survey questions. That’s typically how we work in a production cycle.

GIM: How long is that production cycle in terms of what you do?

DH: Oftentimes we engage with a particular product for a full two years. It depends on whether it’s something quick like an XBLA [XBox Live Arcade] game that that has a much shorter production cycle or if it’s something much longer like a AAA title.

GIM: In doing this research, what discoveries have surprised you?

DH: Oftentimes you get really surprised by little things that people interpret the wrong way or get confused. An example of this is in a game called Toy Soldiers: Cold War. They had this mechanic where you were supposed to get a whole bunch of kills then you would unlock something really awesome and it would be really amazing, but the way that they built the [game] out, it looked like the gun was overheating. Instead of killing a whole bunch of people and getting this awesome reward, people started deliberately avoiding that. It was because the icon that they were using was red [so] they switched it to a blue and made it look a little bit different. There are small moments like that that we see all the time where there’s just a really consistent misinterpretation that if you’ve been working with a game for a really long time, you don’t see because of course you already know what the answer is.

GIM: [Microsoft has recently made some serious improvements in narrative testing.] What have you discovered about how users are connecting with narrative?

DH: We’ve found a couple of consistent patterns. One is when you talk to players about the games that they love most, character really comes through. They have these great relationships with characters; these moments where the narrative goals and the gameplay goals come into conflict and they’ll pick things that are narrative driven because they have a personal relationship with these characters. That’s really powerful and wonderful.

The other thing that we saw is that players have a lot of trouble following plots in games. What we did is asked players to retell the narrative, so tell me from beginning to end the story, your favorite game’s story, whatever they cared about the most. That’s a hard question to judge whether or not somebody’s doing it well, so we also asked them about a story in another media and we looked to see what the differences were between those two things. When we did that, we found that people were really struggling to remember game narratives. There are a lot of reasons why that might be, but for us we began to think about whether it was a comprehension problem. That it was just hard to get all the pieces to fit together into a coherent narrative. We began to look at it like that and see if there was a way to help game developers think about that early in the process when they still had time to fix things. Many of the ways we tell story in games are very expensive to do, like cut scenes and voice-over, things like that. We had sort of a two-part goal. One was to get this improvement in player understanding, but the other is to get that improvement early enough that game teams could actually fix things and respond to issues.

GIM: For new developers who may not have the resources of Microsoft, what do you recommend they think in terms of narrative?

DH: One of the most useful things for a designer to do is to write down the story, not from the sense of what they want from an experience perspective, but rather how are they going to tell it? Really be extremely crisp about that … instead of “Here’s the story that you’re supposed to take away from it,” but rather “Here’s the description of what’s going to be on the screen.” [Then] you can really highlight a lot of areas where people are going to end up confused. … The real thing is about understanding that what the user gets and what a narrative designer needs when they’re building the story are oftentimes a little bit different. Where a narrative designer might build out these wonderful characters with this rich backstory, if you’re not really careful in how you present that to the user, the user may miss out on all that information. It’s really about how you present the information to the user so that they too can have this great narrative experience that you’ve designed for them.

GIM: Were there any particular aspects of character that stood out in terms of what users remembered?

DH: It was less about the character and the importance of the character in the overall story arc, but more about the strength of the characterization. Is everything that the character is doing making perfect sense in the environment that they’re in? Are they interacting with the player in a way that makes sense? It’s something where [characters] weren’t giving you these generic lines, but they were instead giving you scenes that really felt tailored to the experience the player was having. That tended to be when character really, really shone and you got a sense of consistency, a sense that I really know who this is and now I can interact as a person rather than as this [character] construct.

GIM: Can you give me an example of one of those smaller moments?

DH: A great example of this is we asked one participant what one of his favorite game narratives was or a game that had a lot of great story in it. He wanted to talk about Skyrim. We talked about the main story arc and things like that, but when I asked him for a really strong moment from this game, one that really just stood out as just being amazing for him and really, really powerful, he talked about a very, very small moment. He talked about a woman he referred to as “the scary lady,” but when we dug into that a little bit deeper, he talked about this character who was an ambassador who’s throwing a party that you essentially crash. You sort of talk your way in. What I loved about this moment is she is a very minor character but she does everything right. You walk in the door and she does exactly what you’d expect a host to do in that situation, which is she comes up and says, “Sorry, who are you?” and you have to introduce yourself. It’s a moment that just simply works for this player because everything that she did, her tone of voice, the way she approached you, the fact that he couldn’t move… Everything fits together and you’re like, “Oh, no. I’m about to get caught.”

What was great about this is it made the player not just think about the moment as it was happening, but about the moment that would follow. What is going to happen next? And he got very predictive in this. That’s oftentimes what we see when players get really engaged with things. One of the things that happens when they retell stories from other media that we weren’t seeing from games is that they would burst out with this, “Oh, my gosh, I didn’t know what was going to happen next. I was super surprised by that. I was texting my friends. We were all trying to figure out what was going to happen.” Those sorts of moments. And those, I think, are really wonderful and powerful and really demonstrate when you’ve really given someone an extraordinary experience.

GIM: For new developers, what do you recommend they think about in terms of character?

DH: It’s about making sure that your characters are reacting sensibly to the player. When the player makes a decision, it needs to be something that is reflected in the people around them because otherwise it just ends up feeling like a construct. It really is in those tiny, tiny little details where the character acknowledges what the player has just done and talks about it with their own vent, really getting that moment of, “Yes, I know you just did this and I, as this particular character, have this particular take on it.” I think that’s one of the great ways that you can elevate the character. You can move it from being a sort of empty AI to being a person.

Related Content

Have some feedback for our editors? Contact Us