Multimedia
Blogs
Forum
Affordable HD
Whitepapers
Newsletters
Events
Advertisers
Blogcast
MillimeterClean Capture: Making Art on the Motion Capture Frontier
Jan 1, 1998 12:00 PM, Cynthia Wisehart
As the Titanic pulls out of New York harbor, James Cameron wanted a shot that evoked a Royal Caribbean cruise line commercial-a sweeping, ironic viewpoint that begged at least one question: How to populate the deck of the miniature ship? CGI elements would not survive the camera moves, as the shot helicopters over the miniature ocean liner and CGI sea and flies through the billowing digital smokestack. So the deck below bustles with hundreds of digital people, including the avatar of first mate who strides up to join an avatar of Captain Smith (Bernard Hill) at the bridge.
"It had to look absolutely real, and authentic to the period in the way people and their clothes moved," says Andre Bustanoby, motion capture supervisor at Digital Domain. So artists came up with a unique library-based method of motion capture derived through sampling the moves of a handful of performers to create hundreds more.
Using about a dozen costumed actors and shooting at House of Moves, Venice, the performers executed a series of brief vignettes-mother picks up and carries child, man helps woman into chair-all with turn-of-the-century etiquette. Working in Softimage, animators then built hundreds of interactions, scaling height and weight to vary the characters, adding or altering gestures, creating realistic shadows and swinging ponytails. Each snippet was given a number, and later Cameron himself positioned each of the several hundred vignettes on the deck. Sharp-eyed viewers will catch only one point where an avatar passes within several feet of his twin, their walks unmistakably identical.
"We learned that a person's stride is absolutely unique," Bustanoby says. Actually, artists at Digital Domain already knew that from two previous projects. For Michael Jackson's Ghosts video and the Nike virtual Andre Agassi spot, being able to capture every nuance of a performer was the whole point. "A four-year-old looked at the dots for Michael Jackson and said 'that's Michael Jackson,'" says Tom Tolles, of House of Moves, where the Jackson performance capture was shot.
Yet even knowing that, when it came time to create a brief interaction between two recognizable characters-the captain and first mate-the six-foot-one Bustanoby decided to perform the role of the much shorter officer for the capture. "The proportions were all wrong when we saw it," Bustanoby remembers, "and there's a very specific, intense way a military man like that walks. I was more lanky and casual. It wasn't believable that it was the character." (The data was reshoton animator Stephane Couture who studied the actor's movement style and more closely resembled his proportions.)
Animators at ILM made a similar discovery about the specificity of human movement for a motion capture shot of Spawn (in the film of the same name) strolling through flames. Initially, digital effects supervisor Steve Williams performed the walk. "I was surprised at how much of the performer's characteristics ended up in the motion data," says animation supervisor Dennis Turner.
"Steve's data was believable as a character, but not as Michael J. White [the actor who played Spawn]." White was called in for a shoot, but that still left much for animators to do to the data. "We tried to leave the walk itself alone as much as possible," Turner says, noting that hand gestures had to be adjusted, and when Spawn stands and faces the camera, he is completely keyframe at that point.
"It's very difficult to mix hand animation with motion capture data," adds ILM associate visual effects supervisor Habib Zargarpour, "which comes into play when you have to cut back and forth between them. There are so many subtleties to a walk, hip rotation, how weight shifts, things like that. It's hard to get that kind of detail with key frame. It's hard to get the same details." Adds Turner: "As soon as you start fiddling with wrists and elbows, you risk ruining what made the motion capture valuable in the first place which is that complex relationship among parts of the body."
Like most animators, Turner considers motion capture data to be a starting point, in fact the ultimate reference. "Animators are always jumping up to act out something in front of a mirror when they want to get their head around a performance. Motion capture allows us to get the real information of how things work physically. But it's really no more or less valuable to 3-D animation than rotoscoping is to 2-D. When it is used as a reference it enhances the process, but when it is has to be the bulk of the process, it hinders. It is best in situations where you have to have a very specific performance, or where there is no other way to get the data."
On Mortal Kombat: Annihilation, there was no other way to get the data for centaur-like Mataro's battle with Jax. A hybrid of live action and CGI, Mataro was performed by two actors: In the front, Damon McBee was in full costume, including prosthetic legs (and tracking sensors), while the back actor was in the typical blue/sensor outfit. Because the live footage and mocap shoots would run simultaneously, Mataro had to be shot in his environment-on location in Thailand. "A surprisingly smooth shoot," says effects designer William Plant. "We had a skeptical DP [Matt Leonetti], but he helped us immensely; for example, adjusting his lighting to accommodate the video lenses and constructing a kind of motion control rig to insure that we could match his camera moves."
By blending the video tap with the playback from the real-time system, director John Leonetti was able to see and direct the composited character. "That's one of the key benefits of motion capture," says Plant. "You can direct animation like a live-action shoot."
Andrew Tucker, lead supervising animator of the Mortal Kombat team at Digital Magic, Santa Monica, says the biggest hurdle with the Mortal Kombat data was a performance delay between the front and back actors. ("I don't think the back actor was really giving it his all," Tucker speculates. "It can't be much fun to show up to play a main character and find out you're playing the rear end of a horse.") Whatever the reason, the result was that the data had to be slipped in time, with the unpleasant side effect that the model would poke through the live actor if the shift were more than a few frames.
Tucker dismisses the rumored disdain animators have for motion capture, and says he enjoyed finding ways to build on the data. "For example, we observed that if the back end kicked out, the gesture really started at the shoulder, so I built myself a layer of muscle to set up cause and effect." Using the Softimage tool Expressions to link objects, Tucker wrote macros to link various bones and make them perform like muscles. "This somewhat counteracts the tinkering problem because you are applying systems to the animation, and not trying to change it piecemeal. This allowed us to do things like add more kick or more compression in a spring; it was like building in little amplifiers so I could turn up the movement or tone it down.
"Our job is to establish a link between the data and the character," Tucker concludes, identifying an issue at the heart of the mocap/keyframe debate. "You can't just use the data verbatim," agrees ILM animator Turner. "There are things that animators do-enhancing or pulling back for storytelling and character development. The motion capture performer may not always know what the needs of the story will be, so even though a lot of detail is captured it might not be the right details. And the performance is only as good as the person doing the acting."
"So it might be a wonderfully nuanced, unique performance that you couldn't get any other way," counters Digital Domain's Bustanoby, while adding that more often, at least at the high end, it is the key ramers' skill that brings the data to life.
Across the world from Digital Domain, the animation/mocap debate is not raging. At Medialab, Paris, the 26 episodes of Donkey Kong Country were produced for Canadian television with at least 70 percent real-time animation, using Ascension's magnetic system-including the new wireless technology. This show, with its wacky video game characters come to life, represents a different kind of motion capture frontier. Here the technology is not only a new tool, but a whole new aesthetic. Here the art doesn't take place in elaborate post, but in real time, as 23 minutes of cel-like animation are produced in 10 days.
Donkey Kong is on a tight budget, and therefore it is a refined production machine, says David Sturman, head of R&D. Keyframing is the costly spice in the mix. "The process is very disciplined and collaborative," Sturman says. "At every step from script to post we're analyzing the material, making sure we're not setting ourselves up for something impossible." The backbone of the system is Medialab's proprietary Clovis real-time software. Pre and post animation are done in Wavefront/Explore and Softimage, ideally by the same animator at both ends.
Among the limitations they have to deal with, the biggest is problems with contact and grasping; therefore, you will always see Donkey pick things up out of sight. They also avoid side views, since they accentuate any errors in the contact surface between two entities. No more than four characters may be animated in real time to allow for real-time playback, and skeletal constraints of the models mean that Explore and Softimage cannot be used to full capacity. "These technical limitations could be overcome," Sturman says, "but not within the budget we are committed to."
While all this may sound limited, the director also gets to direct his animated characters in real-time, just like a live-action shoot. "So there are compensations," says executive producer Maia Tubiana. "We can really push it on the characters performances and achieve a real depth of expression. It's led to a new type of storytelling that relies on the subtleties motion capture can achieve, and avoids the things it doesn't do well."
"We call it 'enabling' technology, says Dan Smith, CEO of SimGraphics, South Pasadena, referring to the fact that users all over the world can now have cost-effective animated hosts and other goodies for about 1/5 the cost of cel animation. "Accessibility will start to drive the aesthetic," he adds, which will surely continue to develop beyond cyber babes and anthropomorphic comedians.
At the November theme park convention IAAPA, SimGraphics and capture suit manufacturer Polhemus Colchester, Vermont, demoed a completely live straight-to-broadcast performance-a stylish soap opera for two dancers seen on a virtual set. For a live awards show in Malaysia, SimGraphics experimented with pre-rendering an array of poses and moves for a wild and crazy virtual host, while leaving facial expressions to be performed in real time. This meant no body suit was required for broadcast, yet the virtual performer was completely interactive with the audience and fellow performers.
SimGraphics is also perfecting a low-cost system for the O2 with voice-driven lip synch for a character called Fausto, the virtual cousin of Jay Leno's Mr. Brain. Other advances at SimGraphics include filtering to eliminate the classic problem of contact-characters no longer have to sink into the floor to their knees and soon they will not pass their hands right through their dance partners.
One of the most interesting dilemmas of motion capture will continue to be how much of the performance capture data is ultimately used and how much is it altered. In other words: How human should the character look? In the case of a recent Simmons mattress ad captured at House of Moves, the exaggerated "dude-like" skeleton character is funny precisely because his moves are so unmistakably human.
But if a performance capture artist is playing a creature other than human, at what point does the character become the virtual version of a guy in a gorilla suit (remember Trading Places?) The farther the character is from human, the more complicated this becomes, so performers are experimenting with prosthetics and other performance tools for altering the physicality of the characters. Meanwhile, animators and software designers chase the same goal through their disciplines.
At Futurelight, Santa Monica, where the facade of Godzilla secrecy is valiantly maintained, the issue seems particularly timely. Half-serious speculation abounds on whether Futurelight is creating the campy guy-in-a-Godzilla costume look or update the creature to full photorealistic strength. The super-real trailer confirms the answer which was hinted at in a recent demo: Futurelight used its new real-time optical system (based on brain surgery technology five years in the making) to create a Velociraptor that completely obscures its human source with tricks of gravity, weight shifting and an entirely believable tail.
Meanwhile, at the other end of the spectrum are performances in which every human detail is key. This is the thinking that drives Steve Odekerk's collaboration with Venice-based Modern Cartoons. In a project for NBC, a live-action Odekerk (creator of Ace Ventura, The Nutty Professor and Nothing to Lose) interacts with a skeleton, an Odekerkesque avatar, and a cast of splendidly goofy characters from eyeballs to hippopotami to Gumby. The question of who controls this performance-animator or mocap performer-becomes an entanglement full of potential.
Andre Bustanoby likes to wonder if in the future actors and animators will audition for the same roles. In the present, he says he has to break it to producers that motion capture is not always the solution it seems to be. "It's going to be up to the users now to see just what this technology can do, and who knows how far imagination will go?"
Continue the discussion on “Crosstalk” the Millimeter Forum.
