Archiving

[Boomerang]

So I wanted to open a discussion on how everyone is handling there footage.

My initial thoughts are....

Import SD cards as is into two of my internal drives.

Transcode out ProRes to an external for work.
Than Media Manage the final cut on the external and delete unused footage.

This gives to Raw backup version of all the footage and Media Managed version in ProRes on the external.

[PKraft]

Not very efficient, sorry.

Read manual of FCP and how it handles your source material first,
then make assumptions/decisions. Test with small amount of source
and adjust accordingly.


Best Pe.

[Hidef1080]

Purely from an archiving approach I have 3 USB hard drives and I place my PRIVATE folder from the SD card into a ProjectNameofProject folder.
This will go on two USB drives.
My Premiere “finale” edit will stay on my laptop and on the last USB drive.
Once my boss gives me the green light on the “finale” edit I will delete if from my laptop but keep the CS3 on the last USB drive because my boss is the freaking king of “hey, can we go back and do this?” after he says “oh this is great. Just what I needed. Print it!” Yeah, right....


For the record I've never had a hard drive fail on me yet so I feel very good with USB backups and no tape.:thumbup:

[johnnyha]

Boomerang - how big are your two internal drives because I for one like what you're saying in theory. And if you have other drives besides... Hm. If you can occasionally dump that on BluRay...?

[Boomerang]

JohnnyHa - My internals are 1TB each. So roughly 80 hours of original AVCHD footage per drive.
BluRay would be a good way to add safety beyond HD.

As for PE what are some opinions on what would be effiecent.

[johnnyha]

Nice.

[shrigg]

This article (http://www.avchduser.com/articles/avchd_to_bluray.jsp) seems relevent. Anyone have a Blu Ray burner? I don't, and Steve Jobs doesn't want me to have one either (http://www.engadgethd.com/2008/10/14/steve-jobs-calls-blu-ray-a-bag-of-hurt/). :zombie_smiley:

[manglerBMX]

has anyone experimented with breaking up the private folder, storing the contents across dvds, then recombining again? to see if still will work properly?

[shrigg]

With Dual Layer Blu Ray discs already down to around $20 (http://www.e-digital-electronics.com/product_info.php?products_id=124796) Blu Ray could be the way to archive.... One 50Gb Blu Ray would hold 3 16Gb cards plus 2 Gb of relevent project files.... About the same price as HDV tape (remember that?!?). :D

This is sounding better all the time! Good thing Toast 9 supports Blu Ray since Apple doesn't. Does anyone have drive recommendations in IDE? (I want to use it in a firewire case)

[gint12b]

Keep in mind that you can archive 25 minutes of PH Mode on 1 DVD. Or if you have a dual layer burner, you can archive 50 minutes.

This is barely any different from having a 30 minute tape or a 60 minute tape and we have been using that workflow for years...

Just a thought :)

[manglerBMX]

i;ve thought about just shooting to 8gb SDHC cards, have some 16's for longer things, but backing up 8gb would be easy to dual layer dvds. and probably to hard drive as well.

[shrigg]

Keep in mind that you can archive 25 minutes of PH Mode on 1 DVD. Or if you have a dual layer burner, you can archive 50 minutes.

This is barely any different from having a 30 minute tape or a 60 minute tape and we have been using that workflow for years...

Just a thought :)

Well it is the splitting of the AVCHD file system that worries me... I would rather backup complete cards as they were shot without messin with anything

Perhaps a good argument for shooting 8Gb (or even 4Gb) cards....... I like the idea of using my existing burner

[MikeGunter]

I suspect that Blu-ray will be ordinary before too long; then, 25GB and 50GB discs will be the order of the day. By deleting poop, it should be pretty easy to store content on the chips.

[ilauzirika]

What I like about splitting the files into 3.99Gb files (you know, fat32 won't accept more that file size) is great for dvd archiving. Just put your 3.9 file in the dvd along with other smaller one.

[gint12b]

I can't predict the future, but Blu-Ray might just skip over like 8-tracks did and never truly get accepted due to the ridiculous costs of burners, disks, and write speeds...

A dual layer burner is standard on all second generation and beyond macbook pro's and 8GB DL Disks which can hold 45 minutes of PH mode and only set you back about a buck a piece.

Also, an 8GB SDHC Card is only 50 bucks which is nice in case you lose or misplace them. This way you know you will always be fitting exactly 45 minutes on one for archiving.

It all depends on your workflow really...

Blu-Ray technology may eventually come down in price but i fear by the time it does, we will have a better technology to handle our media like 100 GB ethernet internet connection. I can dream right? :)

[MikeGunter]

Yep...

DVDs will still take the max FAT files sizes anyway, and one could put 8GB on a 8GB disc, quid pro quo.

And it's possible Blu-ray won't come down, either. It might hang in that upper cost range. Who knows?

I'm more impressed by the notion of AVCHD everyday, though. I'm saving my pennies.

[Averdahl]

When i buy the camera i will archive the data like this:
1. Shoot with the camera
2. Copy the media to internal drive and edit
3. Copy the media to external drive A
4. Copy the media to external drive B
5. When done editing, export the result and save it to external drive A+B

External drive A will be in my house while external drive B will be at another physical location. Yes, i will swap drive A and B monthly, IE drive A is in my house in january and drive B is at the other physical place. In february i bring drive A to the bank and take home drive B, etc, etc. I will use ~160GB drives, so i will have to buy more than two external drives as well.

Other physical location is *not* in the same house, such as in the attic, in the basement or in another room.

Why all this?:
If external drive A dies i still have external drive B and vice versa. Since i use rather small drives i will not lose too much footage if both drives dies. It may sound cumbersome (and it is) but the day i want to re-use footage or if my house burns down, or if my internal drive dies i will thank myself big time since i will have a drive with the footage on it.

Lots of "ifs", but its better to be safe than sorry IMO! :)

/Roger

[PKraft]

When i buy the camera i will archive the data like this:
1. Shoot with the camera
2. Copy the media to internal drive and edit
3. Copy the media to external drive A
4. Copy the media to external drive B

Roger, I'm absolutely with you. But why copy to a HD and not burn DVDs instead?
They are cheap, cheaper then HDs and they do rarely fail. Burn three (3) of each card
and if you use big cards burn A1, B1, ... A2, B2... A3, B3...
Toast for example does split big cards so that essential files are not chopped into
several pieces but distributed onto separate DVDs.

Security: More then two DVDs never fail at the same time. The remaining DVD is
good to make a one ore two copies of to replace the bad DVDs.

Storage: Store them at a place with CONSTANT LOW temperature (5° C max ),
NO light and constant LOW humidity (less then 55%). That is how big archives work.
No need to re-invent that wheel again.


Pe.

[Averdahl]

But why copy to a HD and not burn DVDs instead?
I get a better overview with HDD's and faster access to the footage when needed when it is on HDD's plus that DVD's have less space. I do loong shootings and will need the large storage space only aviable on HDD's. I am planning to capture the footage via HDMI into my Intensity Pro card and may archive the file/s i capture as well, thus the need for more space than DVD's. :)

I may change my mind though, but what i described earlier is the planned workflow for now though.

/Roger

[PKraft]

I do loong shootings....

I see you point ;-)

However would buy the fastest burner/reader I can afford (cheaper than any HD anyway).
Mind you, this is only an archive copy for safety just in case the internal HD fails. So speed
does not matter that much.

Pe.

[shrigg]

Roger, I'm absolutely with you. But why copy to a HD and not burn DVDs instead?
They are cheap, cheaper then HDs and they do rarely fail. Burn three (3) of each card
and if you use big cards burn A1, B1, ... A2, B2... A3, B3...
Toast for example does split big cards so that essential files are not chopped into
several pieces but distributed onto separate DVDs.

Security: More then two DVDs never fail at the same time. The remaining DVD is
good to make a one ore two copies of to replace the bad DVDs.

Storage: Store them at a place with CONSTANT LOW temperature (5° C max ),
NO light and constant LOW humidity (less then 55%). That is how big archives work.
No need to re-invent that wheel again.


Pe.

Which version of Toast are you using? I'd like to know more about how Toast spans big cards. It does it without modifying the PRIVATE folder structure? I need to test this out..........

Another good suggestion is to burn the A/B/C copies on different media brand/lots.

PS: I have seen pretty serious arguments against hard drives for long term data storage... Check this article (http://www.scottsimmons.tv/blog/2008/09/03/larry-jordans-hard-drive-warning/) out from Larry Jordan

[mcsmooth]

One little tip for PC users is to use the AVCCAM viewer to grab footage off the card. I thought this program was completely useless until I tried its "copy to pc" function. It allows you to copy selected clips over and keep the proper file structure. This way if you have multiple shoots on one card or wanted to split things up into 4/8GB chunks, you can easily do so after the fact without needing to buy smaller cards.

This program is pretty weak, stupid and annoying right now (stop asking me Yes or No!), but could be useful if improved. Otherwise I could see similar simple tools coming about in the future that give you a basic explorer based interface.

[Boomerang]

I agree with the DVD burn but for instance this weekend I may film on 20x 16gig cards....... That many DVDRs would be way to time consuming. Also even if BluRay was mainstream the burn time is so much longer. So for now it is about organization and back up drives.

I have had my share of HD crashes in the past and I am definitely nervous. For the most part this was how I handled P2.

I may even with having two internal backups get a Raid 1 for external editing..... whatever it takes.

[PKraft]

Boom,

for SD Cards, you might even opt to keep those 20 cards for archiving. From a certain production volume upwards, the card costs do not really matter any more.

[ChrisPrine]

One little tip for PC users is to use the AVCCAM viewer to grab footage off the card. I thought this program was completely useless until I tried its "copy to pc" function. It allows you to copy selected clips over and keep the proper file structure. This way if you have multiple shoots on one card or wanted to split things up into 4/8GB chunks, you can easily do so after the fact without needing to buy smaller cards.

This program is pretty weak, stupid and annoying right now (stop asking me Yes or No!), but could be useful if improved. Otherwise I could see similar simple tools coming about in the future that give you a basic explorer based interface.

I can't figure this thing out. It has my 164 clips grouped into 8 different "videos." They are somewhat related. These clips are from 4 different shoots, and I would like to export them based on that. It will have some clips from one shoot and then some others in there.

[Bucknfl]

One little tip for PC users is to use the AVCCAM viewer to grab footage off the card. I thought this program was completely useless until I tried its "copy to pc" function. It allows you to copy selected clips over and keep the proper file structure. This way if you have multiple shoots on one card or wanted to split things up into 4/8GB chunks, you can easily do so after the fact without needing to buy smaller cards.

This program is pretty weak, stupid and annoying right now (stop asking me Yes or No!), but could be useful if improved. Otherwise I could see similar simple tools coming about in the future that give you a basic explorer based interface.

I tried using the avccam viewer on an older laptop but it would only play the video at 10fps and I couldn't see timecode. Could you see timecode when you used the viewer?

[mcsmooth]

I can't figure this thing out. It has my 164 clips grouped into 8 different "videos." They are somewhat related. These clips are from 4 different shoots, and I would like to export them based on that. It will have some clips from one shoot and then some others in there.
Hmm, maybe it has more flaws I haven't run into yet. Do you know if by chance these groups are the playlists? If so, maybe some of the shoots were in the same day? I can't find the documentation that explains how playlists are split up, but remember seeing it in the past. I'm guessing it isn't smart enough to split clips in the same playlist, weak.


I tried using the avccam viewer on an older laptop but it would only play the video at 10fps and I couldn't see timecode. Could you see timecode when you used the viewer?
Let me clarify that I've only found this useful for copying files. It sucks at everything else, and isn't even that great at copying files (but I am not aware of other tools yet). I get 2fps playback, this program does not deserve the title of "viewer". It would help you find which clip has content you are looking for though. I don't think it shows timecode, maybe date/time?

Another limitation for those that want to write to DVD is that they offer the feature of writing the structure to bluray, but not DVD... WHY?!? They make cameras that write to DVDs! One of the benefits of the AVCHD format is that it can be played by bluray players, but bluray burners still too uncommon to not support DVD.

[Averdahl]

(but I am not aware of other tools yet).
Copy and Paste works like a charm! :)

/Roger

[mcsmooth]

^ Kind of a pain if you are trying to copy only a few files over as mentioned above and want to retain the file structure. For the entire folder, I agree. It would have been simpler if the clips were just single files like on a digital camera, but they thought (close-to) bluray compatibility would be more desirable.

[PKraft]

Which version of Toast are you using? I'd like to know more about how Toast spans big cards. It does it without modifying the PRIVATE folder structure? I need to test this out...

Using Toast 9. Does transcode the files I want/need. To check file content and quality, I prefer VideoPier, however.

[Averdahl]

but they thought (close-to) bluray compatibility would be more desirable.
The compability thing is more due to the Mac's file system that limits the file sizes to 4GB than Blu-ray or anything else. :)

/Roger

[PKraft]

The compability thing is more due to the Mac's file system that limits the file sizes to 4GB than Blu-ray or anything else. :)

/Roger
???????
Sorry, never heard or experienced that the MAC's file system does impose any size limits? Must be on another planet ;-)

[mcsmooth]

The compability thing is more due to the Mac's file system that limits the file sizes to 4GB than Blu-ray or anything else. :)
The 4GB limit is a FAT32 limitation, it has nothing to do with macs. I was talking about the file structure, maybe you were thinking file system.

[Averdahl]

Must be on another planet ;-)
No, macs use FAT32 thus no files larger than 4GB. Whos on another palnet, you or me? ;)


I was talking about the file structure, maybe you were thinking file system.
As a wrote, i was thinking of the file system. My bad that i didnt read you post more carefully. :)

/Roger

[trez]

No, macs use FAT32 thus no files larger than 4GB. Whos on another palnet, you or me? ;)

These are Macs from another planet ;) On our planet Macs use HFS and not FAT32...

[BobDiaz]

From Apple's Help WEB Pages...



Summary

Learn the limits for files and volumes on disks that use the Mac OS Extended format (HFS Plus) in Mac OS X, including journaled HFS+ volumes.

Products Affected

Mac OS X Server, Mac OS X 10.4, Mac OS X 10.3, Mac OS X 10.2, Mac OS X 10.1, Mac OS X 10.0, Mac OS X 10.5


Volume and file limits in Mac OS X

Maximum number of volumes (all Mac OS X versions)

no limit

Maximum number of files (or files and folders) in a folder (all Mac OS X versions)

up to 2.1 billion (2)

Maximum volume size and file size (Mac OS X 10.0 - 10.1.5)

2 TB (1)

Maximum volume size and file size (Mac OS X 10.2 - 10.2.8)

8 TB (1)

Maximum volume size and file size (Mac OS X 10.3 - 10.3.9)

16 TB (1)

Maximum volume size and file size (Mac OS X 10.4 or later)

close to 8 EB (1,3)
Notes


The theoretical maximum file size for a Mac OS Extended file system is millions of terabytes. In practice, the maximum file size is equivalent to the maximum volume size, except for a small amount of disk space reserved for file system information.
Specifically, 2^31, or 2,147,483,648. However, the actual number of files that can be stored on a Mac OS Extended (HFS Plus) or journaled HFS+ volume depends on the volume's size and the size of the files. For example, a 160 GB Mac OS Extended volume with the default block size of 4 KB has 40 million available blocks. This volume could store up to 40 million very small files, but not 2 billion. A bigger disk with the same default block size could hold proportionately more files.
2^63 - 2^31 = 9,223,372,034,707,292,160, which is just under 8 exabytes (EB). One exabyte is roughly equivalent to one million terabytes.

For definitions of some of these terms see "Macintosh: File System Specifications & Terms (http://www.info.apple.com/kbnum/n8647)".






Bob Diaz

[Averdahl]

I stand corrected. :)

I have just arrived from planet Coma back to the planet Earth.

/Roger

[PKraft]

Welcome back home. Most importantly, you had a safe journey. ;-)

[Flipper_md]

Hi all

I wondered after trying out a HF10, how to split those monster avchd file to keep only the usefull part, I it got me quite stumped that no native splitter existed.
I filmed in continuous mode while playing guitar, ending up with a 4gig file and quite a few mis-takes in there.

I asked Canon, but got the usual careless answer...go see your software vendor.

I contacted TMPeg team, and they say that the long GOP in avchd make native splitting a problem.

So basically, I would be stuck with a 4gig file to store even if there if only 1/3 of it I want. Or i have to recompress and loose some quality, on top of loosing time to go thru this step... bad.

on the other hand i've read that some low end editors/(dvd softwares?) can do smart render of acvhd files, meaning they do not recompress the data required for output... pretty much what i'd like...

Am i the only one concerned about this splitting problem of avchd?