No Clones: Unexpected Adventures on Episode II

Jun 1, 2002 12:00 PM, by Ellen Wolff

Web-Expanded Version

A Software-Assisted Shoot

Although known for post-production effects expertise, ILM played a key role in facilitating the HD capture process by creating a graphical camera interface that they called the Camera GUI.

“It was essentially a software program written by David Nahman-Ramos that allowed us to set the camera up,” Cooper explains. “We could know, for example, where the black level was set, just in case something got tweaked a little — because it would get tweaked. Instead of chasing our tails wondering why one image was not falling in with the others, we could see how the camera was set up and make adjustments.”

ILM's system was operated discreetly during shooting, hidden inside Fred Meyers' control tent. “We built a portable system that was kept slightly off set,” recalls Meyers, “but it had all the equipment needed to calibrate the color monitors, to put them in an environment where set lighting was not a problem, and also to have control and repeatability. The camera control software allowed us to attach a connector to the camera and know we had all the settings. We were capturing data almost in the same way that we capture data from a film scan. That did require quite a bit of technology. The software controlled the Sony cameras and basically took a snapshot of every single setting. The camera operators didn't have to look at pop-up menus to make sure things were dialed in. They focused their cameras, set their stops, and proceeded as they would on a film-style shoot. The strategy all along was that we would simply replace film technology with HD and proceed as we normally would. If anything, we probably sped up the process, doing 36 setups a day. We didn't have camera problems that slowed us down.”

For scenes where actors rode CG creatures, ILM created digital doubles that replaced all or part of actors’ bodies. New clothing animation software was required to achieve this.

Throughout the first unit shoot, ILM visual effects supervisor John Knoll made close observations of the unique characteristics of the HD images because so many of them would require effects later. Knoll, along with Pablo Helman, Dennis Muren, and Ben Snow, supervised ILM's 2,000-plus effects shots.

“We had more depth of field with the HD. From the production standpoint it was an advantage to shoot in HD because the lighting package could be smaller,” Knoll observes. “But there were surprises. Because the film isn't jumping around in the gate and you don't have all that flicker and grain, the images are super-steady and vibrant. Sometimes we'd be looking at a close-up of an actor and his makeup looked too obvious, or we could see how a set was constructed. Film mushes things together in a way that HD doesn't.”

Instant Gratification

That lesson was reiterated when ILM's miniature and model photography got underway back in Northern California.

“All of us who do this work have gotten used to how things look on film and the level of polish you need to put into something for it to work on screen,” Knoll admits. “It's a bit different in HD. When we were working on our first miniatures, it was a bit of a shock. We had to really clean them up! But since we were seeing them live while we were shooting, we could see what wasn't working. We didn't have to shoot wedges because we were looking at the live feed from the monitor. There were never any questions like ‘Are we going to carry depth of field on that?’ As soon as we wrapped we could take a look. If it was OK, we could strike the set. It was instant gratification.”

Since the HD camera didn't have a time-lapse mode to accumulate long exposures, ILM had to light models a little hotter to shoot in real time. That required building brigher lighting into some models.

Although this capability improved efficiency, there was a downside. “We had to light our miniatures at a level to shoot, essentially, realtime,” Knoll explains. “The HD camera didn't have a time-lapse mode to accumulate a longer exposure, so we had to light a little hotter. That created a problem for any models with internal lighting because we had to build brighter lighting into them. In some cases, we couldn't do the equivalent of stopping down to f/32 to carry depth of field. Even if we could, we couldn't put enough light on the model without burning it. In some cases we'd shoot multiple passes — a foreground focus and a background focus and split them together. It wasn't the absolute ideal system, but I think the advantages outweighed the disadvantages.”

To help with the process, ILM's engineers developed post-processing software that could simulate long exposures. Meyers explains, “Since the cameras have an electronic shutter, in turning off that shutter we could simulate what we called a 360-degree shutter. That allowed us to blend multiple frames together to build exposure, reduce noise, and get shots that could be shot at 12fps or 3fps. Some of the software techniques for manipulating multiple frames had already been well tested. We were able to put those in, although it wasn't simple. We did have to build some processes with the way the tape material gets loaded onto a disk system and then gets pre-processed before it goes into our CG environment. That was all streamlined specifically to handle all the motion-control work. For example, if they decided to do a camera move at 3fps, we had a formula that we would type in, and when that material was loaded it would pre-process that automatically. The other thing that had been done was that we synchronized the HD camera to the motion control system so that we could lock-step the exposure with the camera move.”

Dailies On Demand

To the delight of ILM's effects team, images from the model stage could be projected live in the company's screening room, which was equipped with a Texas Instruments DLP projector from Christie connected to ILM's own custom-built server.

“We implemented systems such that you could be in the theater seeing the image up on the big screen while it was actually being lit on stage,” Meyers says. “The signal coming out of the camera is your image; there's no telecine, no transformation that needs to happen. We took advantage of the fact that digital projection was to the point that we could review material digitally projected without having to go out to film.”

The HD cameras captured so many small details that ILM's models had to be finished to a very precise degree.

Knoll recalls, “One day we were shooting a miniature on stage when George came to the screening room to review some stuff. So we patched a feed from the stage to the big digital projection in the theater and showed it to him. It was a model of a room that had a big scanner in it. George asked, ‘Does that scanner turn?’ So one of coordinators called up the stage, and all of a sudden we saw shadows of people moving in the background. Then the scanner started up. It was hilarious because it was live. We're so unused to that experience. Usually when you're in a theater looking at an image, it's permanent and unchangeable — something that happened long ago instead of happening at that very moment.”

The ability to call up a shot and look at it in on screen with no turnaround time was perhaps the most eye-opening aspect of this project for the creative teams from both ILM and Lucasfilm. “If George's editors wanted to see things in full resolution in a theater they would come here,” Cooper recalls. “They utilized ILM almost as an arm of editorial.”

Take Me To Your Server

The technology that enabled this approach was, to a significant degree, built by ILM's engineers, including the HD conformer and the video servers. ILM also built the databases to track every frame of the movie and sync it up to Lucas' Avid cut.

“What Lucasfilm had in their cutting room,” Meyers explains, “were the down-converts that we had done either in principal or back at ILM, and it was all tied via the time code from the tapes.” ILM's engineers had extended their tools for accepting time code, which allowed them to put a tape into a machine and have it cue up and load whatever shots they wanted to see right onto the server. Whenever ILM added model shots and CG elements to the picture, Meyers notes, “We could tie back into Lucasfilm's editorial and give them the updated takes with the combined elements. We maintained sync between editorial and ILM's elements all the way from start to delivery.”

Handling all this data required huge servers. “We ended up with 14TB in editorial alone,” Cooper admits. “We kept tons of images in full HD online and accessible to our artists. Our video engineering people built an incredible database so that everybody, no matter what platform they were on, could pull up thumbnails of any shot they wanted. When someone made a request, we could pull it off the HD server and deliver it in full resolution. This allowed the visual effects supervisors to see all of their shots in sequence with surrounding shots.”

Animation Director Rob Coleman (foreground) examines the digital version of Yoda with CG Model Supervisor Geoff Campbell.

For Knoll, “It was a great experience to be able to see a shot full-res in dailies, be able to scrub through it and spot all kinds of things that probably would have burned us later if we didn't spot them early on. Our usual process — running low-res for a while then later doing hi-res crops, and then finally running the shot full-res — that process has changed. We started running things high-res much earlier on. Being able see the work at full resolution in the theater has me totally spoiled now. I don't think I can ever go back.”

CG Characters in HD

That's a sentiment echoed by ILM animation director Rob Coleman, who was responsible for the cast of CG characters — including an all-digital Yoda — who populate almost 70 minutes of animation in Episode II. “Having HD servers that can play back high-resolution images at speed has spoiled me too. I was able to see, sooner in the process, what my animators' subtle bits of acting looked like. The twinge of a muscle under an eye or the smallest glance can tell you a lot about a character. But that doesn't show up on the resolution of a 27-inch computer monitor viewed from 18 inches away, and I found I was asking animators to go over the top with their animation. You can create an eye movement that's a couple of pixels on a monitor, but when you blow it up it's a huge eye movement, and you end up with a stylized performance. I was more confident that I was making the correct decisions by seeing it in high resolution. I could make decisions before it was too late.”

Coleman had to make some fine discriminations on the CG in this picture because many of the digital actors were wearing complex, multi-layered costume — often while standing beside real actors. “We needed R&D to rethink how we do clothing. Our clothing simulation team spent months figuring out how to cut and drape digital garments.” The result of this effort was a kind of tweakable simulation software, which Knoll describes as “a simulation engine that could go in and out of preset poses.” Because the movie makes extensive use of digital doubles for stunt shots, he notes, “We needed to be able to cut from a live actor to a CG character and back. So the way a robe draped was constrained at the beginning of some shots by the live-action photography. We had to sculpt that shape and begin our simulation with that — and have it not look awkward. We didn't want to see any seams.”

ILM's improved clothing software was crucial for shots that combined real actors with CG body extensions in the same frame. “We have live-action actors riding CG creatures,” says Knoll. “When we were shooting the bluescreen elements, I had the actors riding a blue shape that was roughly the contours of the creature we would later animate. Rob Coleman and I did the best we could shooting these elements, but it's really, really hard to animate a CG creature that's constrained in its movement — and make it look natural. It's also hard to make the actor look like he's really sitting on the creature and not sliding around. In some cases, it was easier to cut the actor at the waist and replace him with computer graphics from the waist down. Then we could do really tight interactions between his CG legs and the CG creature. In a couple of places the split line moved up higher and ended up being just the actor's head with the rest of the body CG. We used digital doubles a lot, which I hadn't anticipated in advance.”

Surprise And Compromise

Both Knoll and Coleman approached Episode II hoping they would be able to build upon what they'd achieved in Episode I. Both got pleasant surprises, from very different quarters. For Coleman, it was the arrival of computers running the Linux operating system, which presented a departure from ILM's longstanding use of Unix-based computers. “They asked if they could put low-cost Linux machines [Dell P4s] on our animators' desks,” recalls Coleman. “The animators still had their traditional O2s running Unix versions of Softimage and Maya. They could work on either computer. We found that Softimage ran on a Linux machine eight to 10 times faster. So I could see more iterations of each shot because animators could turn things around quicker. I was able to get this movie done faster because of Linux.”

Unexpected for Knoll was the latitude that HD offered Lucas to do one of his favorite things — to select a portion of a frame that he liked, blow it up and reposition it to create a new image.

“One of the things that we wanted to nail down up front was what the limits were in HD of blowing something up,” Knoll recounts. “On Episode I, we found out that for something shot on anamorphic 4-perf we could do about a 17% blowup. For stuff shot on VistaVision we could go up to 30% because of the bigger negative. When we tested HD, we were stunned to find out that almost universally you could do a 30% to 40% blowup. That went completely against my intuition. I thought it would end up being less because of the JPEG compression. I don't have a good explanation for why it held up better than film. I think the noise characteristics are different from film grain. You have to be shooting on a pretty fine grain film to get the noise equivalent of HD.”

Any material that could be captured in HD was, but that didn't include pyro elements. As Coleman observes, “Typically you shoot miniature pyro at a very high frame rate to create a sense of scale because you're comping it to be an enormous pyrotechnic event.” Creating digitally simulated pyro wasn't really an option because, as Coleman observes, “that's still time-consuming to do in computer graphics. John Knoll felt that he needed a real element to achieve the realism that he wanted.” The only alternative, Knoll reports, “was to shoot on film when I needed to run at 300fps. I didn't have an HD camera that could do that.”

The Last Mile

Lucas hopes that Episode II will be shown digitally in as many theatres as possible, but the overwhelming majority of moviegoers will see it projected on film. One of ILM's jobs was to carry out Lucas' mandate that all of those film prints reflect the best quality possible.

To achieve that, explains John Ellis, ILM's manager of imaging services, “We're taking a digital master and creating multiple negatives, and from those the lab will make first-generation prints. Filmmakers typically strike an IP and then, after most prints are made, they'll make maybe six prints off the original negative. This film will have thousands of prints off original negatives. If theaters can't go directly to digital projection, it's the highest quality they can get.”

To create those multiple negatives, ILM took a unique approach. As Knoll recounts, “It occurred to David Tattersall, our DP, that if we were going to take the entire picture through our system, and it all had to start with a digital master and get filmed out, that should we consider doing the conform digitally and then filming out entire reels at once. By doing that multiple times we would make our master printing negatives directly. We all thought ‘Wow, we could do that.’ We'd need to have some new systems in place to do it, but it would be great because we'd save two generations. We started looking into how practical that would be. We'd have to have a method for conforming the whole movie at full HD resolution and do color timing in-house. Then we'd have to do these extremely lengthy film-outs, to ensure there was no color drift. We tested this idea with the movie trailers — every trailer that was seen in a theater was a first-generation print.”

“There are so many advantages to this,” Knoll continues, “that I think it's certainly going to catch on. One of the nice things about conforming it and running the whole thing out ourselves was that we got to control the color timings. I've worked on projects where you make your best effort to get the shots looking the way the director wants, but then later in timing it ends up a stop brighter or redder than you intended. That's heartbreaking because then it's too late. In this case, because the timing was happening here and it was happening so much earlier in the process, if something fell apart there was time to fix it. We ended up with much more consistent color, and more quality control.”

The challenge of filming out entire 2,000-foot reels becomes evident as Ellis explains. “The 30,000 frames we're filming out all have to be perfect. We can't drop a frame because if we do that, the roll isn't viable anymore. We're even filming out the leader so that the lab won't be splicing anything on to these reels. They'll go through printers smoother and faster — it's just a whole new way.”

Filming out more than 1.5 million frames for Episode II puts an exclamation point on what has been an unprecedented production — even by ILM's standards. The impact this approach will have on future projects is yet to be seen of course, but Michael Cooper has some projections. “People may think that this pipeline is only good for HD, but it's simply a digital pipeline. Even if we get a show that requires only 400 visual effects shots, we can scan in the balance of the picture and get it into our full-res conformer — as opposed to filming out the finals and scanning those back in because you need to cut negative. You think ‘Wait, we had a digital final. Why did we go to film with it?’ We could cut it in digitally at full resolution, do the color correction, and then film back out — and give our clients a digital master as well. Even if DPs and directors think digital capture is a bad idea, they like digital intermediates.”

While it will undoubtedly take awhile for other filmmakers to follow George Lucas' lead, Cooper predicts, “They'll quickly get used to having these tools, especially when they realize they will have more control.”

Want to use this article? Click here for options!

BROWSE ISSUES


	DCP November 2008	DCP October 2008	Millimeter Sept/Oct 2008	DCP September 2008	DCP August 2008	Millimeter Jul/Aug 2008

Related Articles

Shoot Tools: Ikegami

Shoot Tools: Thomson Grass Valley

Shoot Tools: Rycote

Shoot Tools: JVC

Shoot Tools: 16x9

Interactive Products & Resources