Realtime Redux
Jan 1, 2003 12:00 PM, By Steve Mullen
Realtime capabilities continue to be a hot issue at both the high and low ends of the NLE market.
 Apple has taken its own approach to realtime with Final Cut Pro 3. Realtime performance will vary, and portions of the timeline will automatically render, depending upon how FCP evaluates the CPU/OS performance limits of the system and the “performance cost” of each effect on the timeline. |
In 2001, when I wrote “Real Needs, Real Questions, Realtime” (see the August 2001 issue of Video Systems), I did not expect to write a second time about the topic. However, at both the high and low ends of the NLE market, realtime continues to be a hot topic. So hot that I have to wonder why.
Is it because no new topic has arisen for the marketing and press symbiosis to talk about? If so, perhaps 24p will take over for realtime in 2003. Or is there a deeper reason for the focus on production speed? When Alan Greenspan talks about increased productivity, he's often referring to advances in technology that increase productivity. Realtime, especially when it costs almost nothing to obtain, is an ideal productivity investment. It's an investment that pays off both during a recession and in the boom times that typically follow.
The question facing those making this investment is: What type of realtime do you want?
I used to think of realtime as simply a positive adjective — like “digital” — that was applied by zealous marketers. But at least digital has a fairly clear meaning. That's not the case with realtime. Both the $60,000 Media 100 844/Xe and Adobe's $600 Premiere 6.5 are touted as realtime, and both are, but in very different ways and using very different technologies. So in what way are products realtime? And how do you match realtime technology with your needs?
One key differentiator is whether the product can output a compressed (DV, DVCPRO25, and soon DVCPRO50) copy of the timeline without delay. Adobe, Apple, and Avid (XDV) offer software-only realtime products that require the timeline to be processed before it can be output via FireWire. For some editors, these products don't qualify as realtime because of this limitation.
Personally, I remain unconvinced that realtime compressed output is a critical requirement for most production. When I hear folks blast a $1,000 product for not delivering instant results, I have to wonder if they have ever worked with film. When a film editor decides that a two-second dissolve should occur between two shots, a laborious process begins. First, 48 additional frames (after the effect starts) are added to the A-roll before a splice is made to black leader. Second, preceding the point where the effect ends, 48 frames of black leader in the B-roll must be cut back to allow 48 frames of the incoming shot to be spliced into it.
The A- and B-rolls are then sent to a lab where they are copied to a print. (This process traditionally required that first the A-roll be printed, then the print-roll be rewound, and the B-roll printed.) If any changes need to be made, the entire physical process must be repeated. This process is not only costly, but it also involves turn-around time that slows the creative process. So for me, what makes realtime so valuable is its ability to support the creative process.
A Little Video History
Effects creation with tape-based video production evolved from film's A-/B-roll model. A-roll and B-roll VTRs, an FX generator box, and a recorder VTR are required. An FX generator must operate in realtime. With linear production, any effect on a 486-line, 720-line, or 1080-line frame must be generated in a frame-time.
The early online NLE systems used the same FX boxes. However, to bring costs down, effects box electronics were squeezed onto a computer board. Special-effect PCI boards have been used by Avid and Media 100 to lower the cost of realtime NLE systems.
The next step was Commodity LSI chips, like the MX-25 and MX-50 from LSI Logic (originally C-Cube). These chips provide a small set of simple, permanently defined, realtime transitions. Pinnacle used the MX-25 chip by itself in the DV500 — to both generate effects and decompress two streams of DV. Matrox used the MX-25 in the RT2000/RT2500 to decompress two DV streams, but used its Flex3D graphics processor to generate effects. The graphics processor approach offered several advantages. Because it's used by gaming software to create realistic 3D graphics, a graphics processor offers very sophisticated and high-quality 2D and 3D video effects. And, unlike the logic within the MX-25/MX-50 chips, the library of effects can be expanded over time.
 On the affordable end of the realtime spectrum is Adobe Premiere 6.5. The application claims realtime capabilities, as do more expensive solutions like Media 100’s 844/Xe. With so many realtime claims, buyers must examine what specific realtime features they need. |
While Matrox and Pinnacle were developing hardware-based solutions, others such as Canopus and Sonic Foundry saw that computer microprocessors were rapidly increasing in power. This computational capability could be used for two tasks. First, it could be used to decompress DV video. Second, it could generate special effects. While this had the potential to lower the cost of realtime products by not requiring special hardware, its true potential lay in the fact that as computer power increased, software-based products could play multiple video streams in realtime. (With today's dual Pentium 4 systems, the upper limit is five streams.)
Those who favor a software approach to realtime FX point out that hardware-based products are typically limited to only two video streams. The hardware camp, while acknowledging this limitation, counters with the argument that software-based products can handle more than two streams only when the effects applied are simple 2D effects or low-quality 3D effects. The hardware camp also argues that an editor is never certain when a timeline will be realtime. As this marketing war is certain to continue well into the future, let's move past it now.
There's no reason for NLE designers to restrict themselves to either hardware or software. Hybrid solutions offer great benefits, as demonstrated by the Matrox RT.X100. A CPU is ideal for decompressing DV because it allows the NLE to be more responsive. A CPU is also ideal for generating certain effects.
For example, most video FX chips work in YUV color space and don't support RGB color alteration. That's why the Matrox DigiSuite's color correction is limited to U and V phase alteration (“hue”) plus saturation adjustment. Other video chips — and most graphics chips — work only in RGB color space, preventing YUV phase adjustments. (Now you know why MX-25 based products don't offer “hue” control.) A CPU can work in both color spaces.
Choose Your Realtime Workflow
Don't let the hype fool you. Making decisions about realtime isn't confined to software-versus-hardware issues. Decisions must be made about other requirements. For example, do you need an NTSC client monitor? Now that many NLE products offer sophisticated color correction, an editor needs to be able to view a full frame on an NTSC monitor. There are several ways an NLE can accomplish this.
For a desktop system, the capture (analog, FireWire, SDI) board likely offers analog and/or SDI output. However, if your NLE uses a PCI card with only FireWire ports, or if the motherboard has a FireWire port, you've got another option. You can drive an NTSC monitor, via S-Video, from some PCI or PCMCIA graphics cards. But here's where you need to ask several questions. Can the graphics card output high-quality video from the RGB overlay? And does your NLE application update the RGB overlay when it is playing the timeline in realtime?
 The Matrox RT.X100 is a good example of a software/hardware hybrid NLE. It uses a CPU to decompress DV and generate certain effects. |
If your board has the latter capability, does your NLE software support it? Laptop users have the same questions to answer about built-in S-Video output.
Laptop owners have discovered that a DV camcorder can be used as a “FireWire-to-analog video” converter. One of the new inexpensive converter boxes can be a cheaper alternative. (Of course, there is no reason desktop owners can't use the same strategy.) But you need to check whether your NLE software will support realtime effects such as color correction while outputting compressed DV via FireWire.
No matter how realtime is accomplished, there remains a major issue. No product can guarantee that every timeline can be played in realtime. Even the Media 100 844/Xe has limitations. If no product can guarantee everything in realtime, what is an acceptable compromise? Do you want the playback of many tracks of any complexity — even if that means the playback will fall below 24/30fps — or even degenerate into severe stuttering? Or would you rather always have perfect playback, even though that means you must render complex timeline segments before you are able to view them?
Avid's Xpress DV 3.5 gives you, the editor, the freedom to make the decision about how you want a timeline to play back. If you can accept the possibility of slower than normal playback, just play the timeline. If the result is too slow or too unsteady to be of value, then render the area. However, by implementing a very fast codec and effects generator, much like the Canopus software, Avid's realtime playback is rarely poor enough to require rendering. Also, Avid has not restricted realtime performance to a subset of effects. All effects have the potential to play back without rendering. Likewise, every effect attribute is available for realtime playback.
Apple has taken a different approach to realtime with Final Cut Pro 3. First, only a small set of effects has the capability to play in realtime. (And not all effect attributes are supported in realtime.) Second, FCP prevents you from playing a timeline that won't play perfectly. When FCP starts, a script is executed to determine CPU type (a G3 rules out realtime), CPU speed, number of CPUs, and the OS (according to the current script, OS X offers less performance than OS9). The script sets a variable (cpuclass) that specifies your Mac's performance limit.
When effects are added to a timeline, each effect is looked up in a table to determine its performance cost. The total cost of all effects is compared to cpuclass. If the cost is less than cpuclass, the frame can be executed in realtime. If the cost exceeds cpuclass, the frame is marked as non-realtime and can't be played without rendering.
If Apple were to provide a way to disable this check, editors could choose to have FCP behave more like XDV. Hopefully Apple will provide this option in Final Cut Pro 4. Apple also needs to work on codec performance so that all attributes of all effects can play without a render.
 Commodity LSI chips, like this MX-25 from LSI Logic (left), are a key component in many realtime products, such as Pinnacle’s DV500 production tools and Matrox’s RT2500 cards. |
In the future, I expect some NLE systems to employ multiple, inexpensive graphics processors as effects generators. These hybrid products — in conjunction with high-bandwidth IEEE 1394b hard disks — will support “many-stream” SD realtime editing. These systems will also support “dual-stream” HD realtime editing. Thus, just as Media 100 has introduced the lower-cost 844/Xi at $25,000, other vendors will introduce higher-priced versions of today's hybrid realtime products.
If I'm correct, unfortunately for those of us who have come to appreciate laptop editing, the gap between laptop and desktop realtime capabilities will widen. Unless, of course, laptops of the future also have access to image engines. These engines could be part of the CPU chip as AltiVec is a part of G4 processors. Alternately, single or multiple engines could be packaged into a chip. This could be accomplished by manufacturers of mobile graphics chips. Or it could be done by a computer company that manufactures laptops and also has a huge stake in the success of laptop editing.
Contributing editor Steve Mullen is owner of Digital Video Consulting, which provides consulting and seminars on digital video technology. Mullen can be reached at d-v-c@mindspring.com. His website is www.mindspring.com/~d-v-c.
feedback
To comment on this article, email the Video Systems editorial staff at vsfeedback@primediabusiness.com.
Continue the discussion on “Crosstalk” the Millimeter Forum.
