• Blogs
• Forums
• Tradeshows
• Partners
• White Papers
• Events

 

Long-distance Relationship

Aug 1, 2006 11:00 AM, By Michael Goldman

Two Different Approaches to Editorial Remote Collaboration

Sidebar
Far-flung Foodfight!

Veterans of long-distance collaboration adventures for editing and general postproduction on major projects in recent years emphasize that the umbrella term “remote collaboration” should rightly be broken down into component parts. The overall concept, they say, really encompasses a series of different requirements that need to be handled and linked simultaneously for true, seamless collaboration to occur over long distances.

Those requirements basically boil down to the challenges of achieving efficient and intuitive data management or information flow, deciding on what type of digital network is needed for realtime or near-realtime communication over great distances, and how to make the whole thing highly secure. Not surprisingly, even within those categories, the needs, demands, and preferences of particular studios, networks, and content creators vary widely. Thus, IP-based solutions are perfectly acceptable for some, but simply out of the question for others who prefer to build their own virtual private networks (VPNs).

Since there is no single, industry-standard approach to these challenges, Millimeter periodically offers snapshots showing how different approaches and component parts are being used on major productions. In this issue, we'll take a look at two recent projects that are currently engaging in extensive remote collaboration for dailies and editing.

The first project, David Fincher's Zodiac, involved a technology growing in popularity in the industry these days: the Internet-based PIX (Project Information Exchange) tool that got attention last year when director Sam Mendes and editor Walter Murch used it on the feature film Jarhead. Millimeter caught up with Zodiac's editor, Angus Wall, and his colleagues for a look at how PIX technology has changed their method of collaboration within the movie's entirely tapeless workflow. (For more on the tapeless workflow designed for Zodiac, see p. 8 of this issue.)

The second project, the HBO series Rome, had different needs, and, therefore, adopted a radically different approach to long-distance editing. At press time, Rome was well into production on its second season in Italy, with all editing taking place in Los Angeles. Christian Wilson, of HBO's advanced technology and operations division, took the time to explain HBO's desire for, and implementation of, a highly secure proprietary VPN for that project — a network designed to permit close-to-realtime collaboration when the new season of Rome entered the editing phase.

Zodiac

In recent months, film-makers integrated PIX in earnest into Zodiac's tapeless workflow. David Fincher insisted the movie be shot directly to hard drives (using Grass Valley Viper FilmStream cameras and D.Mag Digital Film Magazines from S.two of Reno, Nev.), and he also wanted the dailies approach to circumvent tape or DVD at all costs.

Fincher's sound designer, Ren Klyce, and others involved with the production were already intimately familiar with PIX through the company's co-founder, sound editor Eric Dachs, who had assisted Klyce on sound design and editing for Fincher's 2002 release, Panic Room. They also had closely followed Murch's use of the system on Jarhead. Therefore, Fincher and Wall decided early in their preparations to use PIX to communicate throughout the dailies and editing processes.

“Eric [Dachs] was my sound design assistant on Panic Room, and was already developing PIX at that point,” Klyce explains. “At the time, we were mainly concerned about finding a more efficient way to divide up tasks and communicate about them during the [audio] spotting process within the design phase. Eric first created a spotting program on that project to let us assign duties to particular people and then publish the list of who is doing what, and let it live, constantly being updated, on the PIX Internet site so that everyone involved could log in and see the progress of how those duties were being completed. It was very helpful, and kept the project focused in terms of what needed to get done without getting into any redundancy of work by people at different locations. There was this natural cross-pollination that we all liked.

“Then, his company kept developing the service. Jarhead was the first use of the system on a major project in terms of the picture delivery portion of its capabilities, and now virtually everybody is using it on Zodiac.”

PIX was primarily used on Zodiac as “a sort of FTP site on steroids,” according to Wall. The technology was employed as a secure application service provider (ASP) for all dailies viewing and image tracking needs between users in disparate locations. Audio design/editing/mixing approvals, comments and notes on the evolving cut between Fincher and his collaborators at all levels on both sound and picture, efficient management of data and workflow generally, and tracking timelines of evolving work on various shots also required PIX for optimum communication.

Klyce emphasizes that Fincher was the basic hub for applying the technology to the project. He says PIX was primarily about helping the director seamlessly stay in the loop on the evolution of picture and sound edits to aid his creative decision-making process without complicating his life from a technical point of view.

“In terms of audio, which is my area, one frustrating thing for years when working on a film has been the fact that a picture editor might put music or sound onto a version of the cut, while [the sound designer] might be working on the same scene, and showing the director an entirely different set of sounds and dialog and music tracks,” Klyce explains. “PIX became very helpful in fine-tuning the process so that Fincher could see how Angus was evolving the cut and how I was evolving the sound at the same time, so that by the time we arrive at the final edit, Fincher will have heard and seen what we are both doing. The director will have been involved in all the creative options available up until that point. That way, he won't be inundated with having to make a multitude of decisions during the final mix.”

Filmmakers involved with Zodiac say their ability to securely upload imagery and sound to the PIX network, set up permissions for who gets to see what, follow updates to the work in progress through PIX, and communicate with each other through the service's note function was an improvement to their workflow. They add they were satisfied with the level of security on the project, despite the fact that PIX is an Internet-based system. The method of encryption and engineering allows clips to be viewed only in streaming mode, virtually impossible to download, and the filmmakers have eliminated a less secure process from their workflow — the need to bring material to dubbing houses in order to create tapes or DVDs.

But Wall says the ease of communication was, far and away, the part of the service that most improved the editorial team's efficiency.

“During production, [Fincher] was all over California, editorial was in West Hollywood, sound was in Sausalito, and we had two studios in Los Angeles — Paramount and Warner Bros.,” says Wall. “PIX was used to post dailies and edits in HD, and it was great to have one place that everyone could go to access information when it suited them. It was particularly great for sending things to David, as he didn't have to drive an hour to spend five minutes screening something in the editing room. The spotting feature was also of interest, as it essentially allows viewers to add comments that stick to a particular moment in the file. The next time everyone logs on, the notes appear. This is the virtual equivalent of sitting in dailies. And these are just the applications for post. I can only imagine how it can be used to consolidate the sea of information production generates.”

Wall's assistants — Peter Warren, Brian Ufberg, and Wyatt Jones — were responsible for uploading dailies into the system for all to view, using the PIX Uploader tool. They all insist the experience was far simpler from their point of view, according to Warren.

“We don't have to make a bunch of dailies tapes to send out anymore,” Warren says. “In the commercial world, where I'm used to working with Angus, we would often have to make 10 DVDs for clients, four 3/4in. tapes, two VHS, two DVCAM, and so on, and all that is very time consuming, not to mention a huge strain on the environment. Here, using PIX on a show that is not using tape at all, it's simple for us to make one QuickTime [using Apple Final Cut Pro, which is also the platform the movie is being edited on], upload it, and suddenly it's available to all people who need to see it. When we're finished with it, we delete it off the server, and it just disappears — no waste.

“PIX Uploader works very much like an FTP client, but with options that allow you to easily assign who in the production gets access privileges and email alerts. We just make one QuickTime, set privileges and email alerts, upload it, and the rest is automated. PIX emails them automatically when the upload is complete. It's my belief that this system probably saved [the production] from having to hire a third editing assistant, just to make dailies every day.”

Warren emphasizes that, from an editing perspective, bandwidth is not a serious issue with the system, because it is designed so that any approved users with a web browser and a decent Internet connection can participate and view imagery of good enough quality to make editorial decisions.

“We cut at a lower resolution, and we can send out compressed files, but that said, it's still HD and looks far better than many dailies you will see on the other end,” he says. “And besides, think about how many tapes and DVDs we're saving — it's a very progressive system in that sense.”

On a similar note, in a conversation with Millimeter after he wrapped on Jarhead late last year, Murch reflected on where services like PIX are headed, and how such technology might change the filmmaking paradigm. Murch suggests the concept incorporates an idea he has been pursuing since the 1980s — the notion of putting all data necessary for editing picture and/or sound into a central database that is accessible to all approved collaborators.

“I've been keeping a database built in Filemaker Pro on every film I work on for years, so that people can share information,” Murch says. “I think PIX saw that kind of idea as a seed they could combine with the idea of using the high-speed nature of the Internet, plus encryption and other security measures, to deliver images that people could share and discuss in different locations.

“I view this as the start of a postproduction round robin, so to speak, in which the various points of production — film editing, color correction, sound effects, visual effects, music, etc. — all get connected remotely and quickly. Really, the next significant breakthrough will be to take the movie as a collection of data points, and treat those points the way Filemaker treats a file, which is to say, to break down those files into individual records, constantly updating everything. We're now making information available to every one else on the wheel, and removing barriers that used to get in the way. This is the start of all that happening.”

Murch's longtime assistant Sean Cullen says, “[PIX is] a smart technology because it has a file structure — it knows who has seen what things, and who hasn't. It's a huge time-saver and a psychological boost, as well.”

Rome

Still, no matter how smart, efficient, or transparent such communication tools might be, some studios and networks, for various reasons, require thicker pipes, point-to-point contact outside the Internet via private fiber-based networks, direct realtime interaction, and local storage, rather than third-party network storage.

One such project is the HBO series Rome, currently in production on its second season. The project is being shot at Cinecitta Studios in Rome, and, in a major departure from the production methodology used during season one, season two is being edited in Hollywood by an HBO team based at Sunset Gower Studios. During season one, the network built a large editing department at Cinecitta to cut the show, but a detailed analysis led HBO to conclude that the production and editorial departments could collaborate just as efficiently at a lower cost by constructing a sophisticated, highly secure VPN between Rome and Los Angeles.

According to Wilson, HBO built the various elements of this secure, scalable network to satisfy multiple concerns. In particular, producers and directors wanted assurances that they could enjoy the kind of creative give and take with editors that they normally expect during the creative process; all parties involved wanted to be sure they could collaborate remotely and in realtime without risking exposure of their material to the Internet; and HBO needed the process to be more cost-effective than building another editing department at Cinecitta.

Eventually, HBO decided to establish a VPN on the backbone of a terrestrial fiber network supplied by Sohonet of the United Kingdom, a longtime technology partner of HBO, in combination with a variety of other mix-and-match, best-of-breed components pieced strategically together to suit the project's requirements.

“We found there is no one-size-fits-all solution for this type of workflow,” Wilson says. “Some enterprise-class media asset solutions, in the final analysis, were a little constrictive for us, as were the closed network or ‘walled garden’ approach that some major [postproduction vendors] are providing. They offer a variety of bundled services within closed networks, but we found gaps between these offerings and some of the functionality and flexibility this multi-site production required. And, more importantly, we could not allow our digital content to reside on anyone's servers outside of HBO. That meant we also could not adopt solutions from any of the ASP's whose business models require that we co-locate our data on the redundant network servers that they operate.

“Therefore, we had to be progressive and evolve our own arsenal of techniques and modular, interoperable systems to securely exchange encrypted data between sites over an ultra-secure private network.”

HBO thus turned to Sohonet to create an international media production network, according to Wilson. This led to a communication path into which HBO could plug a variety of different tools designed to digitize, encode, encrypt, and transfer dailies direct from a machine room and an Avid Unity system in Rome, through a highly secure network, and into Avid Unity and Media Composer Adrenaline systems in Hollywood, while also allowing robust back-and-forth communication between creative types in Rome and executives and editors in Hollywood about the evolving cut.

It was certainly not inexpensive to adopt this approach, Wilson admits. But compared to building up a massive editing department in Rome, he says, “[It made] a great deal of financial sense, and now, with Rome serving as a proof of concept, I fully expect we'll use this network-based model again on other shows.”

“[Postproduction producer] Todd London could justify the cost by looking at the considerable savings he realizes by [having the show's editing infrastructure] in Los Angeles,” he adds. “It's difficult to quantify the improvements to collaboration and the resulting productivity we derive from this network-based approach. But we can look at the cost of setting up post overseas, including per diem for editors, a large post staff, and so forth, versus setting up in Los Angeles and taking advantage of existing relationships with post houses here, and the proximity to our executive team. Viewed from this perspective, remote-site collaboration via VPN, if done correctly, should pay for itself.”

The basic editing pipeline for season two of Rome, therefore, involves processing and telecining film in Rome, but using Digital Rapids Streamz digital encoding servers and Copper software to both encode (MPEG-2 and Windows Media) and encrypt Avid OMFI files for transmission over Sohonet's fiber network to Los Angeles, where Avid editors can immediately get to work on them. HBO in Los Angeles thus receives dailies the same business day they are telecined in Rome, permitting better management of an enormous, multi-site project.

“A Streamz device, running Copper, receives OMFI from an Avid Media Composer or Unity over a Gigabit LAN in Italy, and it then encrypts that data at 192-bit AES [Advanced Encryption Standard],” Wilson explains. “Rome's Streamz system then contacts its counterpart in Los Angeles, and launches an authentication mechanism, which requires an encrypted key to authenticate every file transfer. If successful, the encrypted transfer is then initiated. For the editor or assistant in Rome, we want the process to be transparent and automated. They need only drag and drop the day's dailies or revised cut into a watch folder on their Avid desktop to transfer it to the Digital Rapids system. The Streamz and Copper will take it from there. Editorial in Hollywood should find, at the start of their day, that dailies are waiting for them, logged and digitized.

“We made a very important leap forward in this regard. The driver in this initiative is a mandate from Bob Zitter, HBO's chief technology officer, to eliminate tape wherever possible from HBO's production, post, delivery, and distribution processes. We're then striving to take the additional step of eliminating physical media, as well, including DVDs or even hard drives, for delivering this media via courier, and we are aggressively adopting a purely network and file-based system as our streamlined delivery model. We handled dailies screeners and some HD scans in this manner for season one of Rome, but we will endeavor to migrate to a primarily network- and file-based FOD [form of delivery] for season two. This is a timely and strategic change, and the advantages in creative collaboration, productivity cost, and efficiency are significant.”

When cuts are ready for review, an Avid bin and a mixed-down mono audio file is then transferred over the same network back to Rome for producers to examine. The other big innovation involved with this workflow, according to Wilson, is the adoption of a technology normally used for other types of applications as a clip viewing/videoconferencing device for editors and filmmakers to communicate in near realtime across the ocean. Wilson says the production is utilizing Sony Ipela videoconferencing systems for this purpose — playing clips directly out of an Avid for realtime viewing on both ends of the network.

The Sony Ipela line includes systems that Wilson refers to as “purpose-built appliances for encoding, encryption, and secure realtime communication” that are normally used in corporate and educational conference room situations.

“We undertook an extensive research, analysis, and due diligence process to find a product that could encode and stream at a full screen, same-as-source frame rate with perfect AV synch over IP at a moderate bit rate [4Mbps], and yet still meet our exacting content security standards,” he explains. “The Sony Ipela offerings are interesting in this regard, because they include realtime AV encoding and streaming devices, and, due to their encryption and authentication capabilities, we discovered we can deploy them for more than videoconferencing by playing sequences directly out of the Avid through the Sony systems, and then piping via VPN the encrypted packets between facilities separated by continents.

“The Avid outputs a single channel of video and two channels of audio, which are fed live into an Ipela unit, compressed in one of a variety of advanced codecs, and encrypted at 128-bit AES. Before a session can be initiated, the send unit contacts the receive unit, requesting authentication via a 32-character key [which is changed every session]. Once designated staff on the receive side authenticates them, the encrypted data packets are sent through uniquely configured firewalls over an external VPN to the receiver. At this point, a robust, realtime, bidirectional collaboration can occur between users at both sites, using mirrored Ipela units. The producer's monitor, rather than residing in the edit bay with the editor, is actually thousands of miles away. It's a good-looking image, somewhat compressed, but with perfect AV synch, so it gets us close to the kind of realtime [long-distance editing] capabilities that we are looking for at a competitive price point. Our selection criteria also required a product line that would eventually support HD. Sony will soon release an HD-SDI capable system, which we are eager to evaluate.”

That approach, combined with the use of desktop teleconferencing tools like Apple iChat and the old-fashioned telephone, permit key personnel to come perilously close to what Wilson says is the ultimate goal for editorial remote collaboration on projects of this nature. That goal: true realtime mirroring of Avid screens and other remote-site postproduction systems, realizing instant, synchronized communication and updates for the evolving edit.

“Think ‘virtual Avid Unity or Apple XSan’ through an ultra-secure, private network,” he suggests, and you will understand the goal that HBO and many others are pursuing in this regard.

“We're not quite there yet, but with advances in three key areas — processor power, network resources, and DRM technologies — we expect to migrate many shows to this distributed production model,” he says. “We realized last year, due to our de-centralized approach [on Rome's first season], with production in Rome, visual effects and ADR in London, mixing and HD finish in Los Angeles, and the promotional campaign in New York, that at nearly any point in the day, somebody, somewhere in the world, was working on Rome. The evolution of this decentralized model will accelerate with the proliferation of emerging interoperability standards, such as MXF.

“We're also looking at other collaborative models, as well, for remote-site review and approval, outside of our trusted networks, via laptop, desktop, or set-top box. But that will require a balance between consumer-ish ease of use, against an unbreakable DRM platform that will offer advanced encryption, authentication and visible and digital watermarking, and forensics features. We need to be certain that files are accessible and viewable by the intended recipient only, so that conditional access privileges will expire, or the system will auto-purge our media after a prescribed window, which is particularly important in the event of loss or theft.”

In pursuit of this goal, Wilson says, HBO is talking to virtually every major ASP company, as well as Avid, about integrating sophisticated collaboration platforms for use within a VPN, rather than through servers operated by those various providers. The products and systems HBO is considering include Sample Digital's digital dailies application products; additional web-based systems from Mesoft and Dax Solutions; and Avid's Transfer Manager, Media Manager, and Interplay software.

He adds that HBO has robust satellite capabilities for various types of long-distance collaboration needs when logistics make setting up terrestrial connections impractical on one end or the other. But in terms of an affordable, realtime, secure solution for the editorial process specifically on most projects, he says the hunt to hone this IP/VPN/DRM approach for realtime applications continues unabated.

“We don't want our Avids exposed to the Internet in any way; nor do we want our material on external servers,” he says. “So, for now, we have instead built our own servers that face each other through various firewalls, multiple layers of security, and encrypted VPN tunnels to handle the exchange of media and metadata that is then pulled into Avids, all running through Sohonet's secure, managed networks . It's not fully a realtime process, though, even though it usually takes just a few seconds to move a list of editorial decisions from one remote system to the next. But the Holy Grail is to be able to mirror the Avids and synchronize them between various postproduction systems and geographically dispersed sites at every stage of production and post.

“There are various systems we are looking at for this, and we have also talked to Avid and others about how to integrate their tools into this kind of network and virtual workspace. There are also systems like [Aspera's] point-to-point file transfer technology that we are interested in. They have written an application to hook into Transfer Manager software so that you could seamlessly publish out of Transfer Manager from your own Media Composer via a background process that also encrypts and accelerates the delivery of metadata and media files between sites. That technology is under evaluation for Rome, and it looks like an interesting solution. And even if not Aspera, then at least that model of a solution to get us closer to mirroring Avids in realtime, while still ensuring our data is safe.”

---

Far-Flung Foodfight!

As it went full-bore into production on its CG-animated feature film Foodfight!, due to be released next year by Lion's Gate, Threshold Animation Studios, Santa Monica, Calif., recently expanded its use of remote collaboration technology that is designed to allow the company to work remotely with individual artists scattered across the globe.

Threshold has been developing the film for several years. During that time, it also developed a strategic partnership with IBM to use IBM IntelliStation workstations to make the movie, and to take advantage of that company's On Demand Computing Center to remotely render shots. (For more on the subject, check out “On-Demand Computing” in the November 2004 Millimeter or at digitalcontentproducer.com/dcc/revfeat/video_ondemand_computing and “Next Generation Digital Studio” in the February 2003 Millimeter or at digitalcontentproducer.com/dcc/revfeat/video_next_generation_digital). Now, the company is using Foodfight! as a test case to help IBM develop a remote digital data management tool that serves as what George Johnsen, Threshold's Chief Technology Officer, calls, in essence, an infrastructure controller. For Foodfight!'s production cycle, the tool has been dubbed the Threshold Effects Asset Manager (TEAM). Johnsen says it's a technology being developed to possibly join IBM's suite of digital content creation tools that operate through the company's WebSphere Portal service.

“[TEAM is a web-based] checkout/check-in asset management system that seamlessly notifies people instantly whenever changes are made that affect them,” Johnsen says. The system includes tools designed to provide a series of ASP-like functions in terms of managing and updating data and communicating changes and important information between Threshold's home base in Santa Monica and an artist's location anywhere in the world.

In terms of how that data gets transmitted, the technology has the potential to be equally useful whether used through an IP-based transmission infrastructure or a specially configured VPN. But Johnsen points out that, while Threshold is microwave-linked direct to a sophisticated fiber hub, it's not using the system to communicate only with collaborators who are also hooked into fiber webs. Rather, to make Foodfight!, Threshold is working with dozens of individual artists in all sorts of remote locations, far from Santa Monica, who have modest broadband connections compared to Threshold's setup. And, he adds, the company is usually transmitting imagery that is of at least HD-level file size, if not bigger, since Foodfight! is a CG-animated feature.

Artists with business-class DSL connections and above have had no problem linking to the system for basic communication tasks, Johnsen says. But he adds, “Speed and connectivity remain the huge issue when it comes to the question of how soon individuals, as opposed to just two or three entities in a point-to-point situation, can fully take advantage of this kind of technology.”

“IBM is very good at compression and security, so we aren't particularly concerned about those issues,” he says. “And the TEAM technology is extremely sophisticated and will work well for us. The problem is that last mile of the connection. We have a good fiber backbone in this country, but because the United States was such an early adopter of this kind of technology, our fiber infrastructure stops early, and the copper end link has started to age. Therefore, we are not in a place yet, like some countries are starting to do, where fiber into individual homes and small businesses is practical. It's my view that the United States needs to get smarter about the distribution of fiber technology going forward.

“Meantime, we already have good delivery of HD-level material at this point. That's great for making editorial decisions, but still at the minimum level for evaluating things like color and texture of animated images. Dedicated fiber and private network providers [like Sohonet and Media.net] are wonderful solutions to this for point-to-point collaboration, but in our case, we are working with remote matte painters, texture artists, and animators who live in faraway places. We work with them because they are fantastic artists and we really like their work, but for now, for some of them, the connection on their end is limited. So, currently, we have amazing collaboration tools from IBM, but the connectivity to some of the people we work with is lagging behind the speed of development of our remote collaboration tools. We'll get there eventually, though.”

In the meantime, for artists with DSL-quality or better connections, Threshold's current methodology involves preloading high-resolution CG shots to its secure site, and once that is done, the company and various artists are capable, according to Johnsen, of communicating in realtime using IBM VOIP tools that are part of its WebSphere package.

“We have voice and picture going back and forth as a steady stream, and we can all talk about it once the shot is up on both ends,” says Johnsen. “That is the only limitation, and it's a bandwidth issue related to viewing a full-resolution shot instantly on both ends. But the point is, once the shot is loaded on both ends, we have direct face time with our key artists, no matter where they are located, and we have sophisticated data management tools to make sure everybody has the right stuff to work with. After all, that's the big thing about remote collaboration beyond bandwidth — managing people remotely to make sure they all have the right pieces to work with. That part of the problem we're solving with the system we're developing with IBM.”
— M.G.

© 2008 Penton Media, Inc.