Skip to content
September 27, 2011 / Randy Coppinger

Media Storage

Drive Interface
I’ve gathered some thoughts about storage issues in a production group environment. But let’s face it, anyone who uses a computer deals with some of these, so cherry pick ideas that improve your digital life.

Version
Remember the last time you collaborated with others on the same document and wondered: Who has the most recent version? How can I know the changes I’ve indicated won’t get lost in the shuffle? This is the issue of version control. The promise of networked storage is that everyone sees the same thing at the same time. But if you are working on more than one drive, or your collaborators “Save As…” often, you spend a lot of time and resources tracking versions. Some software (such as ProTools) forces this issue by allowing only one Write authorization at a time. If anyone else is going to work at the same time, the group makes changes in more than one place at a time. Dividing up the work along clear lines and providing rules for how they get combined are key workflow issues in these cases.

Data Connectors

Fat Pipes

Bandwidth
If your files are low bandwidth — text, pictures, compressed media — then you have lots of great options for networked storage. If your files are high bandwidth — uncompressed audio, video, CAD — then you may find it more cost effective to store files close to you, rather than remotely. Fat pipes are expensive over a long haul. Even the most expensive 3 foot cable is a sunk cost; you pay for it once. Remote storage tends to have ongoing costs, both for connectivity and for the storage. People who work with high bandwidth media files don’t just move them around, we also play them. If you can’t play from your remote storage because of bandwidth limitations then you will copy to local storage and create version control overhead. Some software (ProTools) can’t read/write with low bandwidth storage.

Control
What happens when something goes wrong? If “The Show must go on” then using only remote storage could pose a problem. It’s easier to power cycle a local device, right? Now there are remote storage solutions that feature an army of people smarter than me who can keep the lights on. And that army comes at a cost. But if I don’t backup my data remotely I could have the opposite problem: failure of the stuff right next to me (power outage, etc.) can stop work with no alternative. So some combination of local and remote storage is important. Make sure to decide which storage location is Primary. More often than not primary local storage has helped me meet deadlines because I maintain control of it.

Security
Your personal privacy is one thing. But if you work with data that gives your team an advantage, or losing control of the data would benefit your competitors, then you need security. Local networks have always offered a security advantage. But remote storage solutions have become better and better, to a point where I don’t find this a compelling argument one way or the other. I do think it is important to understand the security context in which you work and take appropriate steps to protect files that need to remain secure.

KeysSearch Keys
Where did I put that project? Naming conventions can go a long way to help you find what you need. If you work with a high volume of projects a clever naming system may still not do the trick. A Search Key is a unique identifier that allows you to differentiate an item. Think Social Security number. I like to use a five or six digit number at the end of every project folder to identify it. All other references to that project — invoices, documentation, correspondence, etc. — include that code so I can go to the correct data set quickly.

Project Life Cycle
In my experience there are different ways we keep files safe from failure by machines and people. Let’s differentiate Safety, Backup and Archive because so often these terms are used interchangeably.

Safety – This is a running copy made while work is underway. For example, during a recording session in years past this might have been a second recorder. These days it’s a redundant array that ensures failure of the primary storage doesn’t nuke the work happening right now. In other words if the primary storage fails, work doesn’t stop.

Backup – At periodic intervals (daily) and/or at significant milestones a copy of the work is put somewhere else. If primary storage fails you’ve only lost work since the last backup. It shouldn’t cause version control problems because you don’t use it for anything but protection. This copy is like a firehouse: you don’t ever want to use it. But when you need to put out a fire, you are very glad to have it.

Archive – Projects become dormant over time. I move files off of active storage after six or more months of inactivity. But that doesn’t mean the client won’t contact me in two years to revisit the material. The process of deep sixing data for use years later is known as archiving. I’ll make the files pretty and try to re-format or re-configure things in an effort to future-proof them. I like to gather all related documentation, screen shots and anything that can help someone use the data again after we’ve all forgotten the particulars, then put that stuff in a master folder. Next I zip the whole thing, apply an MD5 hash to it and copy it all to two different external drives. Every 3-5 years the zipped archive and MD5 hash move to another set of drives.

Follow-up articles: Business Continuity and Disaster Recovery, Media Storage Flavors
See also Rob Schlette’s excellent article 5 Steps for More Dependable Hard Drives at theProAudioFiles.com.

12 Comments

Leave a Comment
  1. Varun Nair (@ntkeep) / Sep 28 2011 8:45 am

    What about future proofing as far as plugin versions are concerned? Do you print stems of everything (even if there are no requirements on the delivery list)?

    • Randy Coppinger / Sep 28 2011 9:59 am

      Good question Varun. I’d like to say that I always flatten the source files (consolidate for the same head and tail on every track), print individual processed tracks, print stems and print alternate mixes (vocal up, instrumental, etc.). I’d also like to say I take screen shots of every plugin. But the truth is I tend to make some subset of those things depending on the project. How about you, Varun?

      • Varun Nair (@ntkeep) / Sep 28 2011 4:20 pm

        As you said, it depends on the project/client. It’s usually consolidation and stems.

  2. kiwiSteve / Sep 28 2011 6:26 pm

    My method is a little spread. I take a multiple backup strategy

    1/ I use Carbonite off site storage to backup my computer in the background. Yep this causes internet bandwidth usage and is not instantaneous. It backs up during idle periods.
    I have selected what folders and files I want backed up (and they are not just Audio but basically everything important to me on my MAC – except apps which I can restore from master DVD’s etc) and it keeps generations offline so I can restore etc as needed. The issue with Carbonite is that the first backup is huge and takes for ever, but once done then only changed files and folders are backed up so the “everyday going forward” hit is moderate. The costs is not that high for space and the cost on internet charges/badwidth is not much AFTER the initial biggie.

    2/ I use Gobbler to backup my Audio & Audio related files – off site.. This is an excellent product and knows about Audio stuff and understands projects etc.. Well it does for the audio apps I use anyway! The issues are similar to the above (1/) but not as bad as only Audio stuff is backed up.

    3/ I use Time- machine on my MAC to backup my MAC to an external usb HD . This is my LOCAL backup copy Works very well and is normally my first port of call for a file or folder restore. Again it runs hourly backups.

    with these above 3 backups I basically have most of my backup needs covered.

    4/ But I still then take a backup of projects onto rewritable DVD’s as projects progress or are completed.

    5/ And I have a portable external USB HD that I also copy a daily (or more often) copy of live projects being worked on.

    All this sounds like a lot of work but 1,2 & 3 are fully automated (once set up which really only took me a few minutes). 4 & 5 I do myself and is reliant on my reliability or nervous disposition to ensure they are done in a timely manner. 4 & 5 are also the ones that take time – my time to do.

    Re versioning.. I never really use any software out there for that but there are a few packages that do it and I understand that maybe the new MAC OSX Lion might have it built in.. Right now I don’t use LION because to many of my critical apps are said to have issues. I also don’t want to be on the bleeding edge of the OS.. I prefer to wait a few bug fix releases before moving.. I heard that Lion has a few bugs and so will watch and wait on that.

    • bobbaffy / Sep 28 2011 6:39 pm

      Steve, any concerns or issues with Gobbler? I just posted on Randy’s G+ about being curious about it, but sounds like you’ve already taken the plunge! Have you had to utilize any of the snapshot/versioning features of it, and if so how did it fare?

      For anyone unfamiliar with the concept of version control software, this is a geeky, but great primer: http://betterexplained.com/articles/a-visual-guide-to-version-control/

    • Randy Coppinger / Sep 28 2011 10:13 pm

      Steve, sounds like Carbonite, Gobbler and Time Machine operate as delayed safeties AND backups at the same time. The DVDs and USB drives are incremental backups you initiate. Why so many versions of each kind of fail-safe… have you lost tons of data from primary AND secondary storage simultaneously?

Trackbacks

  1. Media Storage: Business Continuity and Disaster Recovery « Randy Coppinger
  2. Best of 2011: The Big List of Pro Audio Resources
  3. Media Storage Flavors « Randy Coppinger
  4. The Right Tools for Backup « Randy Coppinger
  5. The Right Tools for Backup « Randy Coppinger

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 50 other followers

%d bloggers like this: