Crossing the Uncanny Valley: Javier von der Pahlen
Javier von der Pahlen may be the first to take us where no one has gone before—beyond the “uncanny valley.” Many have entered the valley, but no one has ever made it to the other side, the place where photorealistic facial animation stops being, well, kind of creepy.
At the 2013 Game Developer’s Conference, Javier von der Pahlen, director of R&D for the Central Studios division of Activision Publishing, and his colleague Jorge Jimenez of Activision Blizzard unveiled their next-generation real-time character rendering. It was jaw-droppingly close to the other side of the valley.
The project was a collaboration between von der Pahlen’s team at Activision and Paul Debevec’s at the University of Southern California Institute for Creative Technologies.
Von der Pahlen, no stranger to crossing frontiers, was born in California but grew up in a half dozen countries in Europe and South America. He spoke to Get In Media about the uncanny valley, his fascination with the human face, and what he looks for when he’s hiring for his cutting-edge team.
Javier von der Pahlen: It began with computer graphics, which I became interested in during the late ’80s before it was taught as a subject. The people of my generation who got into computer graphics came either from a background of engineering or, as in in my case, a background in architecture. I studied at Cornell University, and they had a very strong computer science department that spurred much of the early innovation in computer graphics. That’s how I was exposed. We were doing architectural renders of buildings.
Initially, rendering an image for a single frame was an overnight experience. It was not like what we’re accustomed to now, where hundreds of milliseconds is too long for any impact unless it’s quite groundbreaking. Then in the ’90s you saw Toy Story come around, you saw Jurassic Park, and all of a sudden photorealism seemed to be an achievable goal.
In the beginning of the 2000s, I became interested in faces, so I started researching this with the intention of getting photorealistic faces. I thought, okay, maybe it would take six months to do. And years later I’m still doing it.
JVDP: Think about this—when you first open your eyes when you’re a baby, the first thing you see is the face of your mother. Faces are so present in our lives. It’s the topography we know the best. And we are deeply attuned to it. From an evolutionary point of view, we have to understand faces. Even if we’re not conscious of the fact, we’re all experts in faces. Therefore, they are extremely hard to do, because we’re impossible to fool, because there are seven billion experts out there. I suppose it’s a bit of a quixotic enterprise, trying to reach for the impossible. But it’s not impossible. I think that our GDC presentation proved that not only is it not impossible, but we’re close to getting there in real time, which is to me extremely exciting.
JVDP: We want to cross the uncanny valley, but we’re not all the way across yet. There are things that are still missing and people are working on them. We’re missing hair. We’re missing a certain fleshiness. We have to add the body. In a sense, the reason why a face is such a good starting point is that a face is so complicated and so subtle that a tenth of a millimeter difference will affect its look and make a difference between a mean face and a loving face. It’s just absolutely weird. It’s a very, very subtle thing. But if we can address all the problems in faces, then all these solutions that we find are probably—not directly, but indirectly—applicable to the rest of the environment. Because we have to deal with such a level of subtlety and we have to deal with this in real time.
JVDP: About five years ago, I was lead engineer on Softimage Face Robot, which you guys had early on at Full Sail. We tried to do this automatic face animation, and I realized how deep into the uncanny valley we fell when we got there. So reflecting upon this and looking at the great examples of face animation of that time—that is, Benjamin Button and the work that was being done with Gollum—the question for me was, “Well, these guys are doing a great job. It’s not perfect, but it’s very convincing, so how can I leapfrog them.” And the way to leapfrog them was to go real-time.
Now, the thing about film—and I’m not putting film down in any shape or form, it’s beautiful the work they’re doing—but in film you render a face and a scene and everything from just one camera perspective. That allows you a lot of control and a lot of cheating. If something doesn’t work, just throw a little explosion in there or blur it, change a frame, move a point, paint it. You can do that. But with games we have to be realistic from 360 degrees. The light changes, the perspective changes and it’s even more than that. Those two things will affect a single face, but we don’t have a just single face, we have multiple faces and multiple hours of face animation per game.
So this is an extremely complex enterprise. The presentation at GDC was captured at ITC, compressed with proprietary technology we developed at Activision, and rendered with our proprietary techniques. What’s really, really hard is actually putting everything together. So it’s important to note that I’m not alone in this.
JVDP: All I can say is that we are not making this research just for the love of art. So it will be in our games sometime in the future. I cannot tell you which games we’re working with. But I can tell you, yes, this is not just going to sit on my laptop.
JVDP: My intention is to hire the geniuses out there—and there are geniuses, I’m not just using this word lightly—people who have been looking at the face as long as I have and who are obsessed with one particular aspect of it, whether it be tracking of the pore or rendering the pixel. There are multiple disciplines that go into this. The team that produced this face is about ten or 15 people. And I think the most important quality of every single one of them is their ability to judge the art and understand the engineering and the same time. That is the unique profile I’m searching for.
JVDP: Yes, you have to. The present is the past, you know. If you shoot for anything that exists already, by the time you get there somebody will have shifted the field. And much more so in terms of computer graphics, because there are so many brilliant people working on it.
JVDP: Find something that you excel at. If it’s doing the concept art, the palette, the design, that’s super important. If your inclination is low-level code, we highly value our low-level engineers. You know, it’s a rare breed, the guy or girl that can actually pull an amazing effect out of a few bytes.
So one, follow your passion, and two, find a niche—something that is unique, that hasn’t been done, that you can solve better and faster than anybody else. These are the people that I hire. People that have this ability to find a niche and explore it. When you’re a student, a lot of times you think that all these problems have been solved before. But there really is no problem that has been solved before. Everything is evolving.
Most of the people that I’ve worked with through the last ten years had to convince their supervisors or the companies that something was a worthwhile pursuit. Don’t let your teachers say don’t do that. A lot of times people are not going to see that there’s a fissure or a gap that needs solving or can be solved. So if you see something that you can dive into and solve—do that.
Have some feedback for our editors? Contact Us