• Blogs
• Forums
• Tradeshows
• Partners
• White Papers
• Events

 

Using The Force: How ILM's Army Conquered The Phantom Menace

Jun 1, 1999 12:00 PM, Ellen Wolff

Eight weeks before the premiere of George Lucas' Star Wars Episode 1: The Phantom Menace, things were surprisingly calm at ground zero, Industrial Light + Magic. The visual effects division of Lucas Digital had not yet completed the film's 1,900 effects shots, yet no sense of urgency pervaded ILM's labyrinth-like complex. The only suggestion of stress was the sight of a masseuse arriving with a portable table in tow.

"I'm surprised we've gotten to this point without having a little more commotion," admits Lucas Digital president Jim Morris. "But we worked hard to have our panic attacks long before the movie was due. I had meetings a year ago with [visual effects supervisors] Dennis Muren, John Knoll, and Scott Squires and [animation director] Rob Colemen, during which they said: 'There's no way we're going to get this done!' So we started immediately taking steps to figure out what to do."

The answer was a production pipeline that would enable ILM's 1,000 employees to work efficiently on SW1 (as staffers call it) along with a raft of commercials and several large features including The Mummy, Wild Wild West, and Frankenstein, ILM's first all-CG feature. "Having these big shows, particularly Star Wars, has been a good device to make us build bandwidth," Morris says. "Not that we had any choice. It was the only way for us to crank out work that looked good. The volume of digital work on Star Wars is astonishing--it's really brought to bear the promise of digital."

Fulfilling that promise involved what Dennis Muren has called "the triangle of scanning in film, manipulating it, and getting it out to film in a reasonable amount of time--and on budget." At about $120 million, SW1 has several precedents on the budgetary side, but it's unequaled in terms of technology. Simply put: the entire film--about 2,500 shots-was scanned into ILM's computers and manipulated, making it the largest non-animated digital production ever.

Not that filmmakers initially planned it that way. "Originally, over 200 shots were supposed to be untouched," recalls ILM chief operating officer Gail Currey. "But, of course, after scanning those shots," she says with a laugh, "we decided to touch them." Overall, Currey says that "unbelievable planning" made the massive assignment possible. "George hates waste, so we're constantly questioning whether there's a faster way to do something," she states.

The planning for SW1 began with pre-visualization of every shot in the film. "Lucasfilm did very highly worked-out animatics, which allowed us to see the st ory," Currey explains. "Translating that into information ILM could use was a potential bottleneck, but it was resolved early on."

The Avid Strategy A key strategy revolved around linking ILM via the Avid with Skywalker Ranch, where the film's nonlinear edit was done. Because so many SW1 environments were designed to be digital, ILM FX editor David Tanaka notes: "George couldn't have cut Star Wars on film even if he wanted to. All of the live-action film was processed and immediately telecined to be dealt with on the Avid."

ILM's FX Editorial department then functioned as the "funnel" for the OMF files that flowed from Lucas' editors. ILM actually used three Avid systems-one for each of the film's visual effects supervisors. On Tanaka's computer screen sat icons showing faces of Lucas and ILM's supervisors. "We used their heads to represent the hard drives where we stored their shots," says Tanaka with a smile. "By SW2, maybe their heads will be animated!"

As the edit progressed, files flowed from the Ranch to ILM through a secure fiber-optic network. FX Editorial had to bridge the gap between the original film information and ILM, where everything went through computer graphics. "We reinterpreted the numbers from Avid settings to Silicon Graphics computer settings," explains Tanaka. Pointing to an Avid displaying a typical shot with about eight layers, Tanaka says: "This is like a blueprint. It feels like hieroglyphics sometimes, and we're archeologists decoding these symbols."

Especially critical was the information flow from FX Editorial to the ILM department that scanned the film negative. "I piped key number information over the network, and they scanned the film one-to-one," Tanaka elaborates. "Because we didn't have any workprint, we had to rely on key numbers and couldn't necessarily check footage to make sure it corresponded to what's on the Avid. So we had to come up with another pipeline to check things. Before a scan was sent to CG to be worked on, it was sent here and compared with what was being used in the actual cut of the film."

And new cuts came daily over the network from the Ranch. "We developed a system," Tanaka explains, "where we'd send the data that represented the new cut to all of ILM's Avids. We'd tell them to re-link the media accordingly, which took around 20 minutes. It's a great system that can change things automatically for us, but you still need a point person to 'flag' the difference and convey it to our stages, to CG, and to the art department that's updating storyboards. The golden rule was: If you have a tape dated yesterday, it's useless! For all the rapid speed that the technology produced, we still needed the human component. We had to guarantee that every change in the movie was updated on a daily basis."

As work flowed back from ILM's artists to Lucas for review, FX Editorial became "a funnel in the other direction," says Tanaka. "Since we have communication with SGI computers, the CG department could send us updates of what they were working on. We plugged into every facet of production-like the hub of a wheel with lots of spokes!"

Hyperdrive Scanning When production began, it was evident that the huge volume of scanning would severely tax ILM's single proprietary scanner. "There was concern that scanning would be a bottleneck," recalls ILM software engineer Rod Bogart. "They thought they'd have to run the scanner 24 hours a day to get this vast number of frames through."

Even so, they calculated that they would fall behind. The process of preparing frames involved dirt removal and color timing, as well as an archival process. Bogart notes that scanning is the place where "a paper trail" must begin. "There's two types of data that travel with frames around the system. There's big chunks, like models and other objects. Then there's little chunks, like descriptions that tell us about aspect ratio, color quality, and the film stock something was shot on. We try to have a frame of film appear on people's monitors like it would if they saw it on a lightbox. In scanning it, we can't forget what it was originally."

All of this preparation and documentation takes time. Yet ILM's various departments expected to receive scanned material in a reasonably staged manner. "So two projects went on in parallel," says Bogart. "One was building another scanner. The other was speeding up the one we had. One thing that was a major bottleneck in the old scanner was a piece of hardware that did some special math. We decided to get that out and hook up a high-speed SGI computer to the scanner so we could do processing on a multi-processor machine. We were able to make the scanner about 2.5-times faster than it was. We built the second scanner using the same technology."

"When we got it to run faster by doing the processing quicker," recalls Bogart, "we found that the problem wasn't the hardware. Then we got obsessive, and started asking, 'If instead of 2.5-times faster, what if it were 2.6 times? How many more frames is that a month?' It's a big number! We wanted everything we could squeeze out of it and realized we could rewind faster and ramp up quicker and do a number of other things. When the system was done, we could scan closer to 3.5-times faster than when we started Star Wars." In the end, the scan tally was a half-million frames.

The Virtual Studio Once a week at least, George Lucas drove to ILM from his Skywalker aerie to review artists' work. But most reviews of dailies were "virtual," made possible by electronic transmissions. Over a private network that links Lucasfilm, ILM, Skywalker Sound, and Lucas' home office, Lucas and his collaborators viewed high-resolution images on precisely calibrated monitors. Participants in these virtual meetings stepped through images frame-by-frame or at variable speeds while they discussed the work. They also recorded sessions for future reference.

Fred Meyer, who oversees ILM's transmissions, explains: "Everyone would look at one image that would have not only the dailies material in it but also the images of the people who were participating. Along with high-quality audio, they'd have video pointers so they could point to things on the screen. There was also connectivity between the editors at the Ranch and ILM, so they could set up their own live review sessions to show and talk about how the cut was changed that day."

Transmissions took place in edit suites and mix stages. "We didn't have to send people to a studio," notes Meyer. Even while Lucas was in London for music scoring, Meyer's team set up a three-way transmission with ILM's supervisors and editorial at the Ranch via satellite. "We exploit the locations where people are," he says. "The greater our ability to pull together the principals on a project and move the process forward, the better it is. There are lots of people involved and a lot of data that needs to be moved. The longer data or people are in a holding pattern, the greater the impact."

Accelerating Animation The need to keep the pipeline flowing was most crucial in ILM's computer animation department. With over 800 character-animation shots totaling an hour of screen time, SW1 required a team of 45 animators. One way to facilitate the animation process, especially for scenes depicting armies of Droids, was to use motion capture.

"We've gotten motion capture to the point where it's really viable," states Ken Maruyama, who manages ILM's digital animation group and creature development. "Our optical system is fully tested, even for location work, and our R&D people have developed software that allows us to manage the data in a faster, more automatic way. It saved a tremendous amount of time."

A crucial piece of software also came of age for SW1 to help animators see the fruits of their labors faster than ever before. Called Quick Render, it was developed by Cary Phillips. (Phillips won an Academy Award for ILM's facial animation software, Caricature.)

To put Quick Render in context, Phillips explains: "During The Lost World, an animator would work on animating creatures and then hand off that work to a technical director, who would be responsible for rendering it. Only when animators saw a rendered image would they really be able to evaluate their motion."

Phillips' goal with Quick Render was "to take data spread among different kinds of files and maintained by different people and provide a means for non-technical people to gather it up, put it together, and render a sequence out of it."

"There's several reasons why that's important," he notes. "It used to be that the lighting and animation on a shot would proceed pretty much in tandem, and that complicates scheduling because you've got to have TDs babysitting animators. It also causes TDs to waste time creating lighting setups for motions that could change. So it was a very worthy goal to make a split."

Using Quick Render, animators could render motion tests themselves for Lucas' review. Phillips describes their results as "more than videogame quality."

"George is really good at seeing something and knowing whether it will work when it's eventually rendered with shadows and motion blur," Phillips notes. Once Lucas approved a Quick Render shot, animation director Rob Coleman would stamp it: "Finaled, pending render." Only then would it go on to a TD. This approach was so efficient, Phillips remarks, that only about 10 percent to 15 percent of the shots were sent back to the animators after rendering.

Dailies on the Desktop What is truly striking about SW1 is how much decision-making happened right at artists' desks. The entire facility got a processor upgrade, and disk space and memory were tripled and quadrupled. But it was software innovations that particularly enabled the creative teams to work more efficiently. "Throughput-how many shots we could get through per day and per week-was a big issue," states Christian Rouet, head of ILM's R&D group. "We did a lot of work developing new technology to speed up the process."

Especially productive was a piece of software called Loupe, which uses compression techniques to display a "proof sheet" of shots in sequence on an artist's computer monitor. It allows the artist to click on a shot, watch it run, and also, like a photographer's loupe, "magnify" a portion of the frame for closer inspection.

"It renders out QuickTime motion JPEG movies of individual shots with all the digital elements available," Vince Toscano, Loupe's lead programmer, explains. "People can play back as much material as they want in realtime. Loupe sequences these movies together so artists can put together their own 'edit lists' and visualize how a shot goes from A to B to C. When people can see continuity, it makes a difference in how they do their work."

Toscano continues: "People are using these high-res movie files to actually view dailies at their desks. Supervisors can sit down with individual artists, zoom in on a shot, crop it, and-if there's other material of that shot-do comparisons. They can flip it in and out immediately and see what it would look like, for example, with a little more green." Unlike sitting in a theater, he adds, "They can immediately make decisions."

Because ILM's hundreds of SGI O2 workstations share a file server, "desktop dailies" were widely accessible. Through ILM's web-centric structure, people could look at the production's internal web site, click on thumbnails, and see what was rendered the night before. "Given the ever changing edit from the Ranch, Loupe monitors all the files you open up, so that if someone changes them, it will notify you while you're working," Toscano says. "New material is instantaneously available."

Rouet remarks that as soon as images were done, "we could project them in our theater, straight out of the box."

"Loupe really has revolutionized the process of viewing dailies," Rouet asserts. "It's saved an order of magnitude of efficiency. Because it's much better than video, it replaced video dailies immediately. It even changed the budget for next year's video equipment!"

The Network Glue Twenty-four hours a day, seven days a week, ILM's Gigabit Ethernet provided the connections that held SW1 together through all phases of production. "We've averaged about five terabytes of data a day," says network operations director Raleigh Mann, who joined ILM from AOL. "At peak, we actually pushed 16 terabytes through in a day. Though our network is small in terms of the numbers of computers on it, we maintained a percentage of traffic close to the size of AOL's."

Mann says that e-mail was the number one means of communication on the network. "A huge number of messages go back and forth in a day," he states. "There's no way to coordinate 40 people on the phone, much less in a meeting where it takes people 10 minutes to walk across campus."

The network also became a rendering resource every night during production. "When animators leave, a proprietary system turns their desktops into rendering machines," Mann explains. "So in addition to the massive rendering farm that we have for daytime use, the entire facility becomes one giant computer at night, and it does that over the network. The resource-allocation people figured out how to maximize everything. Nothing remains idle."

After two years in production on SW1, ILM entered the homestretch like a well-oiled machine. Despite having to assemble many sequences with thousands of elements, ILM just "cranked," according to Cliff Plumer, who supervises ILM's TDs. "Meltdowns never really happened. By January, we knew we were going to make it," he states.

"It feels really good that we were able to do this without crisis at the end," Gail Currey admits. "I think Star Wars will cause a light bulb to go off in many people's minds with regard to this way of working." But, she adds with a laugh, "If theaters go to digital projection, the future might be much more frightening-we'll be working up to one minute before screening!"

With scores of CG characters and fantastical sets, The Phantom Menace overflows with impossible realities. What won't be noticeable are live-action scenes that look so much like pure photography that there is no reason to suspect that they are effects shots, too.

Under George Lucas' direction, ILM used its computers to manipulate live-action plates to an unprecedented degree. "We've literally taken parts of four different shots to make one shot," says Lucasfilm's Martin Smith, one of the movie's editors. "We've even changed people's eyeballs to establish eye contact. It's very much directing in post."

One such scene depicts a conversation between the character Padme (Natalie Portman) and the young Anakin Skywalker (Jake Lloyd). Since Lucas preferred each actor's performance in different takes, the shot became a split-screen, composed of elements from two separate takes, says ILM FX Editor David Tanaka.

ILM Compositing Supervisor Greg Maloney created the split by getting the two plates to lock together using proprietary Repo (Repositioning) software. "This scene hadn't been shot with pin-registered cameras, so there was movement in the plates," he recalls. "The procedure I used involved making one shot the 'hero side'-in this case, the Anakin plate-and stabilizing that." After locking that down, Maloney used the information to stabilize the other side.

Making the actors appear to interact convincingly, however, took some finessing. "Padme is first supposed to look at Anakin, then at the floor," notes Tanaka. "But the only footage that was actually photographed was of Padme looking down at the floor first, then looking up at Anakin. So Padme's action was run backwards."

However, the footage of Padme showed steam rising in the background. Once her action was reversed, the steam appeared to be going in the wrong direction. Artists had to perform intricate rotoscoping around Padme so that her performance could be thrown in reverse motion while the steam continued to rise. Padme's hand also passes in front of Anakin, so a rotoscoped holdout matte was needed in order to allow her hand to pass beyond the split onto Anakin's action from the other take. To accomplishthis, Greg Maloney used Avid's Matador for the mattes and ILM's proprietary soft ware Comptime for compositing.

"The historical notion of film editing is that once material has been shot, on the whole, that will dictate the cut," editor Smith observes. "Basically, that's now out the window."

"From now on," Smith laughs, "don't ever believe anything you watch. Anybody who says 'the camera never lies' is a liar!"

© 2008 Penton Media, Inc.