What's the best way to backup original footage from the AF100? I'm new to this type of post production workflow. What steps do you guys take from migrating footage from cards to computer, and how you go about making a duplicate? Someone on here said making a disc image works fairly well. I wanted to get others comments on this.
Best practice of backing up original file footage
- Thread starter swanny
- Start date
What OS are you using?
For most Mac backup of file based media, here is one approach:
http://library.creativecow.net/articles/ross_shane/tapeless-workflow_fcp-7/video-tutorial
For most Mac backup of file based media, here is one approach:
http://library.creativecow.net/articles/ross_shane/tapeless-workflow_fcp-7/video-tutorial
Last edited:
Seth Hampton
Active member
I have always liked BD-R way better than spinning disc. They are getting cheap too. Inkjet BD_R for $1.39 each.
10.6.5
If you're doing it professionally, I'd recommend reading "The DAM book" http://www.thedambook.com/
Although this is principally for stills, the discussion of managing your data workflow and backup regime is also applicable to card-based video shooting.
I have the following basic rules:
1) Write protect the card as soon as it comes out of the camera.
2) Copy the card to a laptop's hard drive using a card reader, copying the whole card file structure (as that may be necessary to interpret the footage- don't drill down and just copy the clips or you might be stuck).
3) Check the footage. On a busy shoot day it may not be practical to look at every clip, but I'd certainly quickly check the first clip and the last clip. Use any old viewer that can read the footage natively (like Panasonic's AVCCAM viewer). You are just trying to ensure that the files are intact.
4) Copy the files to TWO external hard drives. (I like WD passport drives because they are small, come in large capacities and are powered via USB, so no bulky power bricks to take on an aeroplace). You could get by with one external copy if your laptop has a big enough hard drive, but on multiday shoots my laptop fills up so I need the two external drives. Make damn sure you have made multiple backups.
5) At this point with P2 cards, I reset the write protect, put the cards back in a camera and re-formatted. This was always scary, because on set there was often not time to properly check through the footage before we needed to reformat the cards and shoot more stuff.
With SD cards, which are MUCH cheaper, I'm planning to have enough cards to shoot for a full day at least, because that way one can stick to the rule that you never wipe anything off the card until you've checked the footage is OK... just in case you've corrupted something. There are programs which can rescue files from cards if that happens, but not if you've reformatted and recorded over it!
I should note that this is alarmist and paranoid, because in ten years of shooting stills and video digitally I've never had this happen to me. But still, with SD cards being cheap, my maxim will be "don't reformat the cards until you have copies on three physically separate hard drives AND have reviewed the footage".
6) On getting home, you can either choose to rack and stack the location hard drives as your original storage media, or copy to some larger file store system. I have two DroboPro RAID systems (one of which has a spare disk set which I rotate as an off-site backup). One of these is the master copy and I use automated backup jobs to keep the second one as an up-to-date copy. (I wrote some little scripts using rsync for this, but you can get commercial programs to do it if you want a nice user interface). I also copy the files on to the RAID array in my MacPro to actually edit from- following the maxim "at least three copies, one of which is off-site". Actually I'm keeping the location hard drives as well, as disk space is cheap compared with a location reshoot.
I then transcode to ProRes in Final Cut Pro, and have a TimeMachine backup of the RAID array in my MacPro to keep an up-to-date backup of the working files- but I don't figure I need multiple backups of the transcoded footage as I can always reconstitute that from the archived original footage on the DroboPros. I have one directory per project which contains everything pertaining to that project.
When I'm done with a project, I archive everything off to the DroboPro. At this point I am actually still saving the transcoded footage, but I may have to rethink that if I run out of disk space... 16 TB doesn't go that far these days! I also output a finished ProRes render of the whole thing which I regard as my final archival master.
It sounds scary, but it isn't really... the main thing is to make sure that you have at least three copies at all times, and one of them should be off site.
Although this is principally for stills, the discussion of managing your data workflow and backup regime is also applicable to card-based video shooting.
I have the following basic rules:
1) Write protect the card as soon as it comes out of the camera.
2) Copy the card to a laptop's hard drive using a card reader, copying the whole card file structure (as that may be necessary to interpret the footage- don't drill down and just copy the clips or you might be stuck).
3) Check the footage. On a busy shoot day it may not be practical to look at every clip, but I'd certainly quickly check the first clip and the last clip. Use any old viewer that can read the footage natively (like Panasonic's AVCCAM viewer). You are just trying to ensure that the files are intact.
4) Copy the files to TWO external hard drives. (I like WD passport drives because they are small, come in large capacities and are powered via USB, so no bulky power bricks to take on an aeroplace). You could get by with one external copy if your laptop has a big enough hard drive, but on multiday shoots my laptop fills up so I need the two external drives. Make damn sure you have made multiple backups.
5) At this point with P2 cards, I reset the write protect, put the cards back in a camera and re-formatted. This was always scary, because on set there was often not time to properly check through the footage before we needed to reformat the cards and shoot more stuff.
With SD cards, which are MUCH cheaper, I'm planning to have enough cards to shoot for a full day at least, because that way one can stick to the rule that you never wipe anything off the card until you've checked the footage is OK... just in case you've corrupted something. There are programs which can rescue files from cards if that happens, but not if you've reformatted and recorded over it!
I should note that this is alarmist and paranoid, because in ten years of shooting stills and video digitally I've never had this happen to me. But still, with SD cards being cheap, my maxim will be "don't reformat the cards until you have copies on three physically separate hard drives AND have reviewed the footage".
6) On getting home, you can either choose to rack and stack the location hard drives as your original storage media, or copy to some larger file store system. I have two DroboPro RAID systems (one of which has a spare disk set which I rotate as an off-site backup). One of these is the master copy and I use automated backup jobs to keep the second one as an up-to-date copy. (I wrote some little scripts using rsync for this, but you can get commercial programs to do it if you want a nice user interface). I also copy the files on to the RAID array in my MacPro to actually edit from- following the maxim "at least three copies, one of which is off-site". Actually I'm keeping the location hard drives as well, as disk space is cheap compared with a location reshoot.
I then transcode to ProRes in Final Cut Pro, and have a TimeMachine backup of the RAID array in my MacPro to keep an up-to-date backup of the working files- but I don't figure I need multiple backups of the transcoded footage as I can always reconstitute that from the archived original footage on the DroboPros. I have one directory per project which contains everything pertaining to that project.
When I'm done with a project, I archive everything off to the DroboPro. At this point I am actually still saving the transcoded footage, but I may have to rethink that if I run out of disk space... 16 TB doesn't go that far these days! I also output a finished ProRes render of the whole thing which I regard as my final archival master.
It sounds scary, but it isn't really... the main thing is to make sure that you have at least three copies at all times, and one of them should be off site.
Neil Richards
Active member
Copy the "private" top level file directory from your SD card into an archive location on your hard disc/raid array - iMovie supports this nicely by asking you if you want to archive the card but FCP requires you to do it manually. On a Mac you'll see the SD card as a drive called NO NAME and you can just drag and drop the "private" file inside it to wherever you want. The SD card will appear as a drive irrespective of whether you plug the SD card into a reader or connect the camera directly using a USB cable.
Then periodically backup your archive file however you feel is best - time machine, blu-ray discs, tape drive, whatever - preferably to at least two copies, preferably at least one off site
Then periodically backup your archive file however you feel is best - time machine, blu-ray discs, tape drive, whatever - preferably to at least two copies, preferably at least one off site
djkarn105
Well-known member
In addition to what Hywel said, when offloading tape-less media I highly recommend creating a disk image of the original media for a number of reasons.
1)No chance of messing up file structure. Most of the NLE's that ingest the footage rely on the file structure. If you make an image of the entire volume there's no chance of screwing this up.
2)Checksums - most disk image software includes the ability to checksum the image which will verify that there is no data corruption after the transfer.
3) Makes it portable - When you ingest the media from a mounted disk image you can then take that disk image and project file to another computer on another hard drive and re-ingest the footage. It can be finnicky sometimes if you are working with the same source material on different hard drives.
4) Compression - disk images also give you the option of compression which is useful when talking about archiving.
As far as what disk image format to use, I personally use DMG b/c I am on a mac platform but you should theoretically be able to do the same thing on any platform.
1)No chance of messing up file structure. Most of the NLE's that ingest the footage rely on the file structure. If you make an image of the entire volume there's no chance of screwing this up.
2)Checksums - most disk image software includes the ability to checksum the image which will verify that there is no data corruption after the transfer.
3) Makes it portable - When you ingest the media from a mounted disk image you can then take that disk image and project file to another computer on another hard drive and re-ingest the footage. It can be finnicky sometimes if you are working with the same source material on different hard drives.
4) Compression - disk images also give you the option of compression which is useful when talking about archiving.
As far as what disk image format to use, I personally use DMG b/c I am on a mac platform but you should theoretically be able to do the same thing on any platform.
The Drobo looks like a great system. Is that what you use for your Mac pro raid array? I'm looking into setting up a different system than what I'm using now. The drobo looks very versatile.
Are you using disc utility on Mac for this, or something else?
Stu Siegal
Well-known member
Just an fyi, drobo is for archiving only, you can't reliably edit from it.
manglerBMX
Veteran
i always drop the "private" folder onto a folder on an external drive that has a relevant title. when you bring that folder into final cut's L&T that name will be the reel name, thus making it easy for revisiting the project and archiving.
Stu Siegal
Well-known member
If you're not wiping cards in the field, which shouldn't be necessary with the low cost of sd cards, a hard drive archiving alternative is a dual drive dock, Startech or similar, and internal drives. WD 1tb black caviars are cheap, got some for $74/ea at Newegg not long ago.
Oh, that changes things a bit for me. I've been using the raid drives from iomega, but there has got to be a better solution for me.
Rick Burnett
Section Moderator
Just a side question, it seems to take forever to format my SD cards in the AF100, whereas a similar sized CF card in my 7D was literally seconds. Does it take long for all of you as well?
For what it's worth, I've been editing on my Drobo S with an eSATA connection for a few months now with great results. The only issue I've noticed is displaying waveforms in FCP timelines seems to take longer than normal, but other than that all operations work just fine. A few speed tests have confirmed that read and write speeds are just as fast as my internal drives.
EDIT: I should also note that I suspect the waveform issue is due to my choice of WD Green drives, which apparently have unpredictable and frequent spin-downs, translating into an extra few second wait for each drive when the Drobo requests data from them. After that initial delay, things run normally.
EDIT: I should also note that I suspect the waveform issue is due to my choice of WD Green drives, which apparently have unpredictable and frequent spin-downs, translating into an extra few second wait for each drive when the Drobo requests data from them. After that initial delay, things run normally.
Last edited:
What is it about the system that causes it to be less than ideal to edit from?
EidolonFilms
Well-known member
Mine takes longer than it does on the GH1. I'd say it has been taking a minute or two per card on average.
Stu Siegal
Well-known member
Speed. Might work with some projects, but the more stacks/effects, the more likely to choke. It's been pretty widely discussed over on another (bovine) forum, take a quick search.
CraigM
Active member
We have a few of the NewerTech Guardian Maximus external enclosures. They hold 2 hardware mirrored drives in a Raid-1. We buy the empty enclosures and put in our own drives, but you can buy them with drives installed. We also back up older content to Blue Ray.
http://eshop.macsales.com/shop/firewire/usb/raid_1/Gmax
http://eshop.macsales.com/shop/firewire/usb/raid_1/Gmax
djkarn105
Well-known member
Actually, I wrote an apple script that lets me drag the CF card to it and kicks it off automatically but yes it uses disk utility