Beta Sight: Autodesk Discreet-friendly Storage System
Jan 1, 2008 12:00 PM, By Phil Wortas, Director of IT, Creative Group
Blending Dot Hill, QLogic, and Quantum components for scalable, flexible storage.
Creative Group used a robust storage system based on Dot Hill’s 2730 RAID Fibre Channel-to-SAS/SATA storage, several QLogic SANbox 5602 Fibre Channel switches, and a Quantum LTO-3 tape library to handle the Autodesk Discreet workflow it used to create HD graphics packages for ESPN and other clients.
Driven by increasing demand for HD production for ESPN, we at Creative Group recently upgraded our storage infrastructure to keep up.
In order to increase weekly turnaround time for HD jobs, we not only needed more data storage capacity — we needed a more efficient data pipeline to move data along from media ingest to our Autodesk Discreet edit rooms and to archiving.
When the majority of our projects were still SD, the Discreet products would archive the media and project information to a Digi Beta tape. As we started to book more and more HD sessions, we noticed a shortcoming of the Digi Beta archive: Mixed-resolution projects could not be archived to a single tape. Projects would have to be archived to multiple formats — and that created both a cost overhead in stock and an administrative overhead in managing all of the tapes. Kevin Vale, our head Flame/Smoke assistant, and I got to work developing a workflow that would solve our project-management issues.
Archiving the projects to a file seemed to be the best solution, and so we cobbled together an old server, a couple extra Stones we had lying around, and a Sony AIT drive as a proof-of-concept network-attached-storage (NAS) system for performing file archiving. The server was mounted via network-file system (NFS) to all of the Discreet rooms (nine at the time). Kevin then archived the necessary projects to that mount at night. He would then, in a separate step, push the projects off to individual tapes. An internally developed web application would keep track of all of the projects and allow the scheduling department to schedule wires, archives, and deletions.
This system was in place for a while, and it allowed us to successfully keep the HD projects backed up alongside our Digi Beta archives. As mixed-resolution sessions became more the rule than the exception, it became obvious that we'd need a much more robust system to keep up with the sheer amount of data that was being generated daily. That's when we contacted Jeff Elterman of DigitalMix to upgrade our storage infrastructure. Elterman made recommendations to enable us to archive projects to near-line SATA disks and then to LTO-3 tape, as well as edit video on SAS disks — allowing us to leverage the new features of Discreet's 2008 upgrade. Discreet's 2008 upgrade finally repeals the “Stone Tax” by introducing StandardFS as a method for using a standard file system that can live on much less expensive third-party hardware than Discreet Stone storage. Also, the 2008 upgrade enables Discreet Backdraft to perform background archiving and wiring, which enables our archivist to work during the day, rather than starting his shift at the end of sessions.
We decided to incrementally implement Elterman's recommendations by first purchasing significant disk storage based on Dot Hill's 2730 RAID Fibre Channel-to-SAS/SATA RAID storage, several QLogic SANbox 5602 Fibre Channel switches, and a Quantum LTO-3 tape library. After we put the new storage online, it showed an immediate effect: Our once-crowded Stones, regularly running at 80 percent of capacity, quickly dropped to only active projects. And our archivist now works the day shift. We are in the process of completing Elterman's recommendations with the installation of a SAN based on Quantum (formerly ADIC) StorNext FS to further leverage the repealed Stone Tax to the max. (Prior to the Discreet 2008 upgrade, the only SAN solution Discreet supported was Autodesk's proprietary and prohibitively expensive Stone Shared SAN solution.) At a fraction of the cost of the Stone Shared SAN solution, the Quantum-based SAN solution will increase disk utilization by consolidating storage for our entire creative team facility wide. The SAN will save time and disk space by eliminating wire transfers of enormous amounts of HD project data from local Stone storage, as well as the transfer of graphic files from local servers used by graphic artists to video editors.
Elterman recommended the Dot Hill storage because it delivers high sustained data rates, the key benchmark for video streaming at a low cost per terabyte. There are two RAID controller implementations unique to the Dot Hill storage that are particularly well suited for video storage. One is SimulCache, a technology that instantly and simultaneously mirrors cache between RAID controllers while the controllers are in active/active mode and delivering maximum bandwidth. The other is EcoStor, a technology eliminating the need for battery cache backup during a power loss. These new technologies distinguish Dot Hill's RAID storage from the common implementation employed by most manufacturers, which preserve data in cache during power loss with a battery, and they mirror cache data by writing data from one controller's cache to the other controller's cache in an active/active dual-RAID configuration. One drawback to the common cache-mirroring implementation is that in order to maximize bandwidth, dual controllers must run in independent mode, which results in giving up cache mirroring and the ability to preserve data in cache in the event of a power loss. A drawback of the battery backup for cache is that in the event of a power failure, depending on how long it takes to restore power, the controllers will disable write cache-back mode and operate with significantly reduced bandwidth until the batteries in each of the controllers are recharged. It often takes hours for batteries to recharge, resulting in unnecessary downtime before video capture can resume.
Fibre Channel-to-SAS/SATA RAID storage has proven to be very cost-effective for us. We populate disk enclosures with a mix of lower-cost SATA drives for near-line storage, for our Backdraft machine to archive projects, and higher-bandwidth SAS drives for streaming video. More expensive enclosures containing RAID controllers are daisy-chained to JBOD (just a bunch of disks) enclosures without controllers. As we scale up capacity and bandwidth, the number of enclosures with and without controllers — as well as the number of SATA and SAS drives — are incrementally added to precisely fit our requirements. At the end of the day, this storage solution has enabled us to dramatically increase weekly HD production turnaround time. And we have a cost-effective road map to increase data capacity and bandwidth as we grow our business.
Creative Group is a diversified entertainment and media company dedicated to the creative arts, producing live action for television and film, commercials, documentaries, visual effects, broadcast design, and branding on behalf of clients such as ESPN, CBS, Viacom, Disney, and Nickelodeon, as well as for some of the top agencies in New York. For more information, visit www.creativegroup.tv.
DigitalMix provides storage and network consultation, sales, and service for leading video post and broadcast companies in New York. Contact Jeff Elterman of DigitalMix at (516) 922-2221.
