Karl Paulsen /
08.01.2011 11:05 AM
Preparing for Data Migration, Part 2

We continue from last month's introduction on data migration. We surmised that as organizations employing media evolve, they find their storage environment becoming inadequate to support their workflows. Sometimes their first approach is to add more storage; in other cases they look to replacing the storage platform all together.

When facilities have a modest amount of physical storage, a moderate degree of work activity or employ only simple direct attached storage (DAS); moving the data to new storage won't present much of a challenge logistically or financially. However, when you reach the scale of a-few-to-many hundreds of terabytes of storage, or upwards of 50 or more concurrent users, these decisions take on entirely different perspectives.


Fueled by collaborative working environments, enormous quantities or increased sizes of files, data migration today encompasses a broader set of considerations than it ever had to five years ago. When data migration happens, it usually includes a technology refresh aimed at providing efficiencies as well as capacity increases or platform changes. This opportunity doesn't come often, and as such can be fraught with risks if all the aspects are not considered.

Fig. 1: Possible migration environments that facilities may face as they grow or production and distribution needs expand.
Fig. 1 depicts various migration environments that facilities may face as their businesses grow or as their production and distribution needs expand. It should be evident that these changes will affect varying levels of the system whenever one looks at moving data to augmented storage, when replacing storage altogether or when changing operating systems and hardware platforms.

As these media-centric systems expand, not only does the program-like content expand, so do the databases that support them. It is important to understand not only the audio-video nature of these files, but the interrelationships that these systems have with metadata, clips lists, EDLs, business processes and the like.

One of the checks often overlooked is how storage is managed for efficiency. This should be examined long before deciding to wholesale replace your storage platform simply because you've found it lacks needed capacity. Be certain your current workflow practices are not exacerbating the storage capacity issues. Take a look at storage management practices that can alleviate extraneous and duplicated sets of large files. Understand how versioning of your content impacts duplicative file sets. Look at places where sets of clips differ in context by only a few seconds, such as a tag at the end or a graphic overlay in the middle.

If there is a separate file for each version in rough cut form, it is likely there are multiple copies of the "final" release version also being kept. Often production workflows create multiple copies of the same content and don't realize it, taxing storage systems tremendously. There are software tools that can help manage these inefficiencies and should be considered as part of the solution to any storage update.


If you're going to replace your current storage system, don't migrate your old, often bad, practices. Bad habits are generally created because of poor system management where at first the installation initially looked fine, but over time short cuts occurred and data management suffered. Data migration is one of those times when you should fix the issues that might have contributed to the reasoning behind having to replace, rather than simply augment, your current storage environment.

Remediate orphaned storage and look at how to gain efficiencies, optimize performance and make the best use of the new storage platform. Leverage those tasks that can be automated to get rid of the manual processes previously used in data migration, production processes or system reconciliation. Look for tools that accelerate the movement of the data. Such tools and practices will help not only in the current migration, but also for future migrations as storage continues to expand over time.

The tasks of managing risks (such as security leaks and extended downtime) should be part of the migration plan. Care should be exercised when migrating data among varying arrays provided by differing vendors. This process usually finds that permissions and security settings (either temporary or legacy) could be left behind, making your data open to potential theft or corruption.

If you're changing operating systems, moving data among file systems (e.g., NTFS to NFS) could also result in the loss of permission and security settings that remain obscured to the casual user, but quite obvious to the skilled hacker. On another level, permissions could be lost entirely, preventing access to migrated data in the future.

Make certain a complete backup to an archive system is made before the process of data migration begins. Using an archive provides a level of insulation that could allow for data recovery at a later time.

Don't neglect the importance of providing for sufficient training on the new procedures, software and systems. Use all the ramp up and transition time to get everyone familiar with the new systems. Be certain you've provided enough time for debugging prior to abandoning the previous storage infrastructures.

Lastly, look at the total picture. Start at the front end of the process and move through each and every component clear to the back end. Data migration needs to be viewed as an end-to-end solution, encompassing the network infrastructure, software and the workflow models so as to achieve the best value from the new components.

Karl Paulsen, CPBE, is a SMPTE Fellow, technologist and consultant to the digital media and entertainment industry with Diversified Systems. Contact him at kpaulsen@divsystems.com.

Post New Comment
If you are already a member, or would like to receive email alerts as new comments are
made, please login or register.

Enter the code shown above:

(Note: If you cannot read the numbers in the above
image, reload the page to generate a new one.)

Posted by: Anonymous
Tue, 03-06-2012 01:15 AM Report Comment
Thanks for the stream of sinscooucness on HPC storage. I'm currently trying to solve a large storage challenge for one of my clusters. Do you have a suggestion for 8000 IOPS and 50TB real? I'm looking to grow to over 100TB. In terms of filesystem, GPFS and Lustre are top contenders. Cheers.
Posted by: Anonymous
Fri, 08-03-2012 11:50 AM Report Comment
I needed extra space for vioeds and DVDs and didn't want to spend a high amount of money for what local stores offered. This seemed like the a cheap, easy solution. It is what it is pictured to be, a cheap metal rack that hangs on a door. It only holds VCR tapes in the cardboard sleeves and still lays flat against the door. If you put in DVD cases or the hard plastic VCR tape cases it causes it to stick out more. Still for the money, it serves it's purpose. It also rattles against the door if you move the door. If you want something perfect, you have to pay more.

Thursday 11:07 AM
The Best Deconstruction of a 4K Shoot You'll Ever Read
With higher resolutions and larger HD screens, wide shots using very wide lenses can be a problem because they allow viewers to see that infinity doesn’t quite resolve into perfect sharpness.

Featured Articles
Exhibitions & Events
Discover TV Technology