Game Changer: Lead Analyst Darius Kazemi
His first gig had him testing weapons for Dungeons and Dragons Online, but since then Blue Fang Games' lead analyst has become an authority on automation and metrics.
If you want to break into the gaming industry, become a game tester. Of all the modern memes associated with launching a career in gaming, this one tops the list by a Crysis 2-sized power jump. More visitors to Get in Media inquire about this particular topic than any other. Heck, Sony, a company that’s not exactly known for having a wild-‘n’-crazy side, even built two reasonably successful seasons of a reality TV show on it. (And foisted Big Fazeek and War Princess on us. Thanks, guys.)
That’s all well and good, but it begs the question—is the Tao of the Tester meme or myth? Does the four-lane superhighway to game-industry greatness actually run through quality assurance?
Darius Kazemi certainly thinks so—in fact, he’s living proof that, with opportunity and a lot of hard work, it’s possible to build a career that begins with basic bug-crunching. Six years ago, Kazemi got his start as a game tester with Boston-based Turbine, wading through the weapons sets of Lord of the Rings Online. In addition to headaches and bouts of blurred vision, it gave him valuable experience, connections and opportunities that eventually helped him form his own company. Today, Kazemi is known as a trailblazer and expert in the use of metrics in social and MMO gaming—he’s the lead analyst for Boston-based Blue Fang Games, makers of the Zoo Tycoon series, Zoo Kingdom, and forthcoming Facebook-focused reboots of PC classics The Oregon Trail and Where in the World is Carmen Sandiego? He’s also the author of the game-industry blog Tiny Subversions, where he opines on all things game development.
As it turns out, Kazemi’s experience is a combination of meme-building and mythbusting. You can break into industry as a game tester, but anyone who thinks the job’s a simple matter of getting paid to play games all day has a few thousand more virtual weapons to swing.
Get In Media: People, including gaming big wheelers like David Jaffe, often talk about quality assurance, being a tester, as being this illustrious entrée into great things in the gaming industry. Do you think that’s still true? Was it for you?
Darius Kazemi: It was definitely true for me. Certainly 10 years ago, the No. 1 way to get into the game industry in any position was through QA. That’s less true today. Now that there are so many game schools and people graduating with degrees and things like that. Plus, game companies are more mature as well, in terms of hiring practices and so forth. Nowadays, getting into the industry through QA is definitely one option, but there are other equally viable options, especially if you have art or programming skills. If you have a computer science degree and you’re really good, and you have an impressive portfolio of games that you’ve made, then you can get a junior programmer position with a games company. Ten years ago? That would have been a lot harder. You would have had to start in QA.
‘QA is not playing games all day. In fact, the things you look for in a good QA tester are good communication skills.’
I knew people who started in QA, then let their bosses know that they had programming skills—they were like technical QA people, and they started working with programmers and would get promoted internally. That was how it used to work all the time, including in 2005 when I started. You would really take a QA job and weasel your way into whatever you could get.
GIM: Some still hold onto this impression that the basic qualifications to become a QA tester don’t rise much higher than being an expert controller monkey/couch potato.
DK: It certainly wasn’t the case five years ago when I was in QA. There are people who get fired from QA all the time, so it’s not an easy job. In a lot of ways, it’s extremely brutal. The way I like to put it to people is, you know, QA is often seen as “Hey, you get paid to play games.” It’s sort of like, if I walk up to you and say, ‘Hey, do you like soccer? I’ll pay you $10 an hour to play soccer all day.” And then you show up to work, and I hand you 10,000 soccer balls and ask you to measure each one to make sure that they’re the correct circumference and record the data in this spreadsheet. That’s what QA is. It can be a repetitive job. We had thousands of weapons in D&D Online. I remember one time, I had to sit there and type in the command to spawn each weapon, equip it, swing the weapon, make sure the weapon- swinging animation worked correctly and then type in the next command to spawn the next weapon. I had 1,000 weapons to go through.
GIM: At what point did your vision begin to blur?
DK: My eyes blurred a lot. It’s that kind of work that made me want to get into automation. No person should have to do this—this should be a job for a computer. QA is not playing games all day. In fact, the things you look for in a good QA tester are good communication skills. You’re playing through a portion of the game, and usually you’re following a test plan and the test plan says, load up the level, walk from here to here, make sure this door opens, make sure it shuts, make sure you can’t walk through the door while it’s closed, make sure you can’t walk through the door while it’s open. All that kind of stuff. Following this test plan, if you get to a point where it turns out that if the door is in its closed state, and you can walk through the door like it’s not really there, then you have to write a bug report.
The bug report is the most important thing that a QA tester narrates. You have to explain concisely but clearly what the problem is, and you have to give steps to reproduce it—load up the game, teleport to this location in the game and open the door and walk through the door. It has to be a recipe, so anyone who reads it can attempt to reproduce the same bug you just found. That report goes to the developers, and the developers load up their copy of the game and make sure it really is a bug. The next thing they might do is ask you a few questions about how you got it, maybe you were a specific type of character at the time. Generating these QA reports is the most important thing that a QA tester does. So really they look for good writers—at least I do. I want someone who can write or communicate. A QA tester does not have to be good at playing video games. In fact, if they’re the kind of person who tends to break every video game they touch, you’re probably a better candidate for a QA tester. Skill level is irrelevant. What really matters is your ability to communicate, be organized, pay attention to detail and your fortitude to do repetitive things a lot.
GIM: Take me back to 2005, and walk me through your own experience.
DK: When I graduated from college, I knew I wanted to be in the game industry, and I knew I wanted to be in Boston, so I sort of applied to a bunch of different companies in the Boston area—probably 12 different companies. I didn’t hear back from almost any of them, but the last one I applied to was Turbine. It was the last one I applied to because I don’t like MMORPGs. (chuckles) Just not my favorite kind of game, and that’s of course what they specialize in. But they called me back, and the main reason I got that callback was that I had known a QA tester there for about 3 years from going to local events. He put in a good word for me, and I got a callback, and I got a QA gig. It was a standard, 10-dollar-an-hour, poorly paid QA job. I was working on Dungeons and Dragons Online. And it was really exciting, my first job out of college working for a private company.
I was a QA tester for 6 months, and during that time I got more involved in automation and metrics, and I was promoted to data analyst. I stayed on in that position for another year or so. In March 2007, I left and started my own company.
GIM: In terms of where you are today, was getting into metrics and analysis a career turning point?
DK: Yeah, that first six months at Turbine really kind of set the stage for the rest of my career. I had some programming skill—I had graduated with an engineering degree and was an avid hobby programmer—so while I was a QA tester, I got more involved in what’s called automation, where you program the game to play for you, so you don’t have to do the repetitive crap so much. I did a lot of automation tasks on Excel—you can actually automate Excel to generate stuff. Automation led into metrics. I found out that we had a metric database at Turbine for D&D Online, and I didn’t see very much use coming out of it. I thought, “Hey, we could track down some bugs using this database. We could look at our live players in our beta servers and analyze the data and track down bugs.” That’s how I pitched it to my boss, and he said “Sure, see what you can do in a week.” I spent a week, and I came up with all these reports, and people were pretty impressed with that. Then it was a matter of maybe 6-8 weeks before I was promoted to full-time data analyst. I was mostly working on Lord of the Rings Online for the rest of that year. I was designing and implementing metric systems.
GIM: Looking at what’s happened in MMORPGS and the social network gaming space, it seems like you were pretty far ahead of the game.
DK: I like to think so. When I started this, I was one of the only people doing it. The only other guy who was making a lot of noise about this was Larry Mellon. Larry is now a friend and colleague, and we work together on projects. At the time, I think it was 2002-2003, he gave a talk at the Game Developers Conference in San Jose. He was working on the Sims Online, and he gave a talk about all their metrics and automation systems. To this day, that talk is still remembered by people as being influential. There were people like me who were toiling in metrics obscurity, so I’m not saying it was me and Larry and nobody else. But when I started speaking at conferences about metrics, Larry was the only other one. It felt kind of lonely.
‘In my years of working as a metrics person in both MMOs and social games, one of the things I’ve seen over and over again is that a good game designer’s intuitions are usually correct.’
GIM: Expand on that point a little more. Why do you think it’s important for companies to be paying attention to metrics? Where’s the benefit? Where does it go off the rails?
DK: When you have a game that you sell in a box, and it’s a one-off product, all you really care about is whether people buy that box. You put it on the shelves, and you just want to make sure people buy it. You pay a lot of attention to a big advertising campaign and building up a lot of hype and making the game high quality, so you get good reviews. But most of the game’s sales are in the first month for a box product. For an MMO or a social/Facebook game, the game is ongoing. It lasts, in theory, forever. You have the unique opportunity to collect information about your players and make decisions about how to retain your players based on that information. Games have moved from being a product in a box to more of a service model.
Because you’re working with a service model, you need to take those metrics, and you need to pay attention to them. At this point, I don’t think there’s anyone working in online games who sees metrics as a bad thing or a waste of time. The real debate is where you apply the metrics—what’s the appropriate use? No one is going to say you shouldn’t pay attention to the number of people playing your game—that’s a no-brainer. The real arguments start coming when it comes to questions of how you apply metrics to game design. Think about a studio like Zynga, where they’re extremely metrics-focused in their design. If tweaking a small thing increases revenue by .01 percent, and the amount of money to do it was outpaced by the amount of revenue it generates, then that tweak was a good thing. Then there’d be another tweak. They do a lot of A/B testing. And a lot people don’t like it. I don’t like it myself.
GIM: I think I know where you’re going with this, but go ahead and elaborate.
DK: In my years of working as a metrics person in both MMOs and social games, one of the things I’ve seen over and over again is that a good game designer’s intuitions are usually correct. As a metrics person, you’re there to show them the small percentage of times they’re not correct and say, “Hey, something you made was wrong, and here’s why.” Or, “People aren’t behaving or responding the way you expect them to.” I really like to work very closely with game designers—and really, the whole team— on figuring out which metrics to implement and how to interpret them.
For me, it’s really a holistic process, and I don’t believe that driving your entire game design through metrics is a good solution. There are a lot of dangers to doing that. Like if you set your goal for the amount of time people spend in your game to be between 10 and 30 minutes, and then you see people who are spending two hours in the game, you might think that’s bad for some reason—it’s an outlier. It’s not what you expected, and you’re thinking something’s wrong with the design. Is the player stuck? Are they unable to read a screen? You need to investigate that, not immediately squash it. People might be spending two hours in the game because they’re taking videos, or maybe they’re choreographing a big dance routine. If your instinct is to just take all the curves and normalize them, you could miss out on a lot of the fun that happens in the outliers of those curves. I’m very much about collecting a lot of data, coming up with the right questions to ask around it, and then using that data to investigate further and further.
GIM: I think we can agree that metrics has exploded at this point. Here’s a trickier angle: Where do you see it going next?
DK: I just see more and more positions for analysts opening up. I would love to see game schools start to teach this kind of stuff. I’ve already spoken with at least one program, and I’ll hopefully be advising them on a university math/game analytics type of thing. I see it growing. There is a big backlash against metrics, but I think it’ll go back and forth. It’ll be a debate that‘ll happen forever, similar to the debate we’ve had in games whether game design should be technology-driven or design-driven. I know there are people putting together books on analytics for games. It’ll be more codified, and best practices will be put out. Just like any new type of position, it’ll start becoming more common. Just like we all know what we’re looking for when we want to hire a particular 3-D graphics programmer, in a few years, people will know what they’re looking for when they’re trying to hire an analyst.
Have some feedback for our editors? Contact Us