Help Explain The Storage Issue!

[Barry_Green]

I think the rough part right now is the camera price is declining (HVX) but the storage solutions have not....yet.

Again, I do not follow. Storage prices are plummeting every day. Hard disks are cutting their costs dramatically; it seems like every month they're bigger and cheaper than they were last month. Already hard disks cost less than DVCPRO-HD tape, and DVDs are amazingly cheap as well, and will continue to get cheaper.

 

[dop16mm]

You don't need a raid to archive your footage, perhaps to edit but not to archive. Archival hardrives can be in simple removable trays that you put on a shelf. cheap 3.5" IDE drives will do. DVCProHD tapes are not cheap, about a dollar a GB. Shooting P2 and dumping straight to hd would be no more expensive and possibly less money than shooting on DVCProHD tape if it were an option. Comparing directly with JVC means shooting 720/24p which only uses 40Mb/s when capturing on P2 before harddrive transfer, 2.5 times more footage than at the full 100Mb, less than half of what it would cost to shoot on HD tape.

This is my planned workflow.


-Build a external firewire or usb2 box with removable drive tray.
-prepare drives in trays according to how much footage is needed for a project. (like buying tapes)
-on set have laptop and external drive for dumping P2.
-When drive is full, pull out and label just like a tape and put in new one.
-In the edit bay plug in your external drive holder. This is now your DVCProHD deck (cost about $50)
-Put one of your full disks (tape) into the drive (deck) and view footage, transfer selects to your main edit drives, and put your master disk (tape) back on the shelf.

You don't have to have all of your footage online all the time, you can always transfer from your masters just like you would with tape. I am more confident in the durability of a hard drive than a cheap dv tape. When your project is locked your can either back up master drives to something cheap like dvd-r, and re-use drives or keep as masters or whatever you like. If you re-use tapes you'll probably re-use drives.

 

[Barry_Green]

Let's go back to the beginning and see what we can do...

Hi Oliver, thanks for that but I guess I was wondering if one could break down the following three cameras options for storage and what the pros and cons are? All the HDV, DVCPro, etc. is making me dizzy and not sure which one of those formats is on tape, disc, requires other hardware, etc.

Panasonic HVX200

It shoots DV to miniDV tape, and it records DVCPRO50 and high-def to its P2 cards. It also provides the ability to record through firewire to numerous devices: you can choose to record on an external DVCPRO50 or DVCPRO-HD deck, onto a laptop or desktop running a capture application, or onto a FireStore type of hard disk. The most important factor when considering the HVX is what it records, not necessarily what it records onto. It records DVCPRO-HD, a professional high-def codec as used in the $70,000 VariCam. It has frame-discrete compression, with higher color sampling and no interframe artifacting as found on the competition. What you choose to record it on is, largely, up to you. The camera provides P2 as an onboard option, and streaming through the firewire port for external capture by whatever means you choose. The key thing is, what it outputs is always full-quality DVCPRO-HD.

Recording via P2 eliminates any risk of dropouts or dropped frames, and eliminates the need to capture or digitize your footage -- you can edit straight from the card, or execute a high-speed copy operation to copy the contents to your computer's hard disks.

The HVX offers 1080i, 1080p, and 720p high-def acquisition, including variable frame rates up to 720/60p.

Canon XL H1

The XL H1 provides onboard capturing using HDV tape. HDV video is not in the same class as DVCPRO-HD. It's very highly compressed, with low color sampling and the tape-based recording format leaves it open for potential issues such as 15-frame dropouts. It's also a linear acquisition model that requires you to spend time digitizing footage, and have hard disk space adequate to the task.

HDV video is very space efficient. The Canon's 1080i HDV format uses 1/4 the space of DVCPRO-HD 1080i system.

The Canon provides two means for recording onto external equipment -- you can record through the firewire port, or the HD-SDI jack. Recording through firewire means you will still have HDV compression, but at least it will eliminate any dropout risk and will also eliminate any need to "capture" the footage afterwards.

The HD-SDI port offers the potential for the highest-caliber capture, if only you have a system to capture the data. HD-SDI capture is not cheap. At the bare minimum it would require something like a RAID of 6 or 8 hard disks, probably at least $15,000 worth of equipment, and probably a desktop computer. Certainly not a portable field-based acquisition system; however, for studio use, if you have access to that level of gear, you could get very high quality indeed. But it'll take up space -- you get about six seconds per gigabyte when recording uncompressed HD-SDI high-def.

The Canon offers 1080i, and also 1080F recording (which we don't know exactly what that means, other than it is claimed to be the equivalent of 1080p). The Canon offers no provision for 720p acquisition or recording.

JVC HD100

The JVC offers a similar workflow to the Canon. It offers HDV recording to tape or via firewire (although using the 720P variant of HDV, not 1080i). It also has analog component output (as does the HVX and XL H1), which could theoretically be used to capture uncompressed high-quality video like the HD-SDI port (but all the other restrictions apply, you'd need a similar RAID and computer setup to handle capturing analog component HD or HD-SDI). HD-SDI should provide a better signal to capture than analog component would.

The JVC's HDV mode takes up less space, it's 19 megabits per second, so about 20% less space requirement as compared to 1080i HDV, and about 1/5 the storage requirements of DVCPRO-HD. It also uses MPEG-2 compression with low color sampling (4:2:0) and an interframe codec.

------------------------
Okay, so now that that's all done, let's get rid of the concept of capturing analog component or HD-SDI, because frankly nobody here making a micro-budget indie film is ever going to do that. You're looking at either capturing HDV on tape or Firestore, or DVCPRO-HD on P2 or FireStore. That's the real world and what you're going to have to work with.

In that comparison, the FireStore solution is available to both, and offers similar workflow benefits. The HDV version will give you more-compressed, lower-color-sampled footage but more of it per gigabyte. The DVCPRO-HD version will give you higher-quality, higher-color-sampled footage but take up more space.

In a P2 vs. Tape comparison, many people have their opinions; I think P2 is a vastly superior acquisition medium, but it's limited in duration. You get instant access, instant review, instant deletion, pre-record, and instant editing, dropout-free, condensation-free, shockproof, heatproof, coldproof, dustproof, etc. And you can edit it immediately. Vs. tape, which offers none of those benefits and also needs to be digitized; but the tape can in and of itself serve as an archival copy, and individual tapes are cheap.

 

[stephenlnoe]

There are many alternatives. You can store it wherever you like. You can store it on DVDs or on blu-ray or HD-DVD or on DLT or LTO tape or on hard disks or on DVCPRO-HD tape...

There is no disputing that. The fact is that there is no given storage method built in. The P2 (or firestore for that matter) user must prepair a way to archive (or just delete) the data. It is what it is.

In fact, if you wanted to get silly about it, you could always archive it onto HDV tape too. I mean, if you're satisfied enough with the quality of HDV to trust it to shoot your original footage, surely you'd think highly enough of it to use it to archive your footage, right? I wouldn't, I'm just saying it's a possibility.

There is no disputing this possibility as well, however, it is more money thrown at the archiving issue. Panasonic is clear about their intended workflow for P2. It does include an Archive server (data storage) and a Material server (index database). The Material server could be placed on the NLE computer but the Archive server (or HDD space) is not accounted for in the workflow. My understanding is that Panasonic will allow download of "P2 Viewer" software that allows viewing and metadata fields to be altered. This could act (possibly) as the Material server front end pointing to where-ever the files are stored.

Maybe. Maybe not. Maybe you want to master to your delivery codec. Maybe you want to master to uncompressed for some reason, or to a visually lossless codec. Not sure what you're getting at there...

Of couse you want to master to the highest possible quality for your footage which usually is the native codec. If your supplying an EDL to a post house with the original footage then they can master it of couse. If you're doing it in house it only makes sense to master to the timeline codec you are working in. But you're right, there are alternatives.

As you know, with HDV you just send it back out to tape and call it a night (probably a late one). I remember, B.Green, that you are against tape completely. Maybe you can see that it does serve a very important purpose. That's why it's been used for around 60 years to back up computer data. And that's what were talking about here, not images but digital computer data.

 

[braw]

A big terabyte raid could cost an extra $5,000 - $10,000.

How big are you talking here? I just priced a 1.2 terabyte array for just over $1000 in raid 5. That would give you enough space for just under 20 hours of DVCPRO HD.

That's using 4 400GB drives. That's way more than enough for a feature, and has redundant backup. I personally think that's the best bet. Permanent storage may be an issue, but unless something massively catastrophic happened, you won't loose that footage. When you have your final cut, sort what footage you want to preserve and rent a deck for a day and dump it to tape. That way you only use what tape you actually need for the footage you want. Or, when other large storage option come along, rent or buy that option.

While it is quite a backward version of what we're use to, I think it forces a much more streamlined way of working with footage.

Just my 2 cents.

 

[SquidLips]

Excellent Barry!

Thank you for really reading into what I was asking. That cleared up about everything I needed to know. It seems that Firestore system is going to be a simultaneous buy for me. My guess is I will be using this camera mainly for the 720 format to be able to use the variable frames but would be nice to have the extra safety of not relying on tape with the Firestore.

Good job everyone.

 

[stephenlnoe]

@braw-I have seen DASD ultraSCSI gigabit raids for $5-8000 You can move way up the food chain for great bandwidth on multiple streams of data. This comes in handy for multiple clips on the timeline. I wish I had it!

What solution did you find for $1000? FW? Network drive? SCSI?


@dop16mm- You are right on. Archiving and online editing are two different animals. The SCSI solution could be for archive and online edit depending on the pipe it is connected to the NLE with. Your solution with the removeable drives seems well though out and could be viable and sensible. Have you tried a dry run with anything? Is this something you have in place now?


This scenario is very similar to our audio solution. Here is the current audio setup:

Laptop with Sonar 4 PE
Edirol UA-101
ADS external FW drive enclosure(s)
Power strip to support 3 devices
Cables
Mics
(occasionally) Mixing board

All of this travels in it's own trunk. I can see the P2 scenario being very similar to the audio setup we currently use in that there are drives involved and a laptop and programs that run to recieve the data. I know it well. Once the drive is brought back in house then it is plugged into the DAW and mixed accordingly and then mated to the video later on down the line. IT IS A PAIN!!!! So much so that we've talked about just getting a 4 track or 6 track box and leaving it at that. The audio is pristine and great but most of the time I wonder if it's worth it (the hassle).

 

[braw]

What solution did you find for $1000? FW? Network drive? SCSI?

A promise raid card and 4 400GB SATA150 drives. SCSI in my personal opinion are overkill. RAID 5 gives you redundancy and striping at the same time. DVCPRO HD isn't really that insane to require an insane hard drive. Toms Hardware did a raid 5 breakdown, and at 4 drives the throughput is so much more than sufficient on paper.

I would never recommend prebuilt external arrays as they are quite a bit more than building one yourself. Or if you have the money and not the time, then get a prebuilt deal. Unless you plan on using a laptop, or need to move the footage between locations, why not internalize it to your main machine? You don't need a terabyte on location to dump footage unless you're doing uncompressed. At that point, why even question this, you should have mass financial resources.

I assumed all internal setup to my machine. I would probably use P2 and an external single drive to dump on for the day, then transfer to the home base when I get home. Or get a Firestore and do the same. I would archive for myself however it works for what I'm doing. If I had to turn over footage, I would guess DVCPRO tape or just get one hard drive at whatever size needed for how much footage I captured.

For example, one 300GB drive is about $150 and could store 5 hours of DVCPRO HD footage. You can get smaller if theres less to hand over.

Check New Egg or your favorite comp store and check these articles:

Raid 5 Scaling
Raid 5 In Windows XP

 

[Boomerang]

My POV really is when you shoot miniDV its archived on the spot as long as you keep your tape. When dealing with P2 you have to have a work flow that is mindful of your raw footage. I wouldnt go so far as to say storage prices are "plummeting". I cant really find a 4.7 gig DVD a useful storage solution when dealing with 20+ hours of documentry footage and interviews.

P2 is not a bad thing by any means, it just is a new way to view your workflow. Its exciting to see what advantages we will reep in the future dealing in a tapeless world.

 

[Nathyn]

How big are you talking here? I just priced a 1.2 terabyte array for just over $1000 in raid 5. That would give you enough space for just under 20 hours of DVCPRO HD.

That's using 4 400GB drives. That's way more than enough for a feature, and has redundant backup. I personally think that's the best bet. Permanent storage may be an issue, but unless something massively catastrophic happened, you won't loose that footage. When you have your final cut, sort what footage you want to preserve and rent a deck for a day and dump it to tape. That way you only use what tape you actually need for the footage you want. Or, when other large storage option come along, rent or buy that option.

While it is quite a backward version of what we're use to, I think it forces a much more streamlined way of working with footage.

Just my 2 cents.

You're right. I was screwing up my my math. As a matter of fact I was looking at a terabyte harddrive to edit HDV from but now that Barry showed me where I was screwing up at a have a whole new out look on this. The HVX is looking real good to me. I still like the idea of storing three hours of info on an 80 gig Firestore. I looked at some stills from the JVC and they looked good, but I'd really like to shoot 1080i/24p. I really want a shoulder mounted camera but the HVX is looking really good right now. I think I could shoot 720/24p and be happy. I'd get about 2 hours or so on the FS. I'm not really worried about archieval storage but on site and storage for editing. These are my big concerns about HD.

-Nate

 

[goldyprog]

Excellent thread. I want to fill you all in as to what I have in mind regarding an HVX. Per my pre-production for a documentary, I want to use the HVX overseas, and use 720, 24p. The problem, of course, is having storage. With 16 GB in the P2, according to Panasonic, that allows me for 40 minutes of continuous storage/recording time. The solution I have in mind:

-Labtop, such as an iBook, which will serve as an outlet to dump the P2 DVCProHD to harddrive

-Gtech, 1 terrabyte of external storage, for about 1200 USD at B&H (maybe lower if one researches. I currently have a G-Tech with 320 GB, and that amounts to 306 GB after its operating storage is considered. I'll bring this along for extra storage. According to my box, approximately 1.75 hours (or 105 minutes) of DVCProHD is stored per 100 GB. They do not specify 1080 or 720. So, if I use about 986 GB + 306 GB External harddrive connected to a labtop via a 6 pin to 9 pin firewire, that is approximately 1290 GB of storage, which is approximately 1350 minutes, which equals roughly 22 and a half hours. My original g-tech320 drive was about $380. Buying a $1000 labtop and $1200 for 1 terrabyte of storage will serve my needs, and capture native DVCProHD, according to my calculations. I am aiming for lots of footage, and don't want to afford all this video data being lost by a hardware glitch. Thus, as a secondary option, I will convert the P2 data to tape on the HVX as well, just in case. It's best to have it at least in downconversion to SD than to not have the footage at all, in case the native footage is lost on the G-tech. I don't know if you guys agree, but this is valuable if you are away from a workstation for a while, and need to capture hours of footage, and as native DVCProHD.

I hope I'm right. Any discrepancies, especially the G-tech quote on the DVCProHD and whether they may be referring to just 720 or just 1080, holler! Thanks! otherwise, if there is a less expensive external drive supporting native DVCProHD . . .

 

[SquidLips]

That seems to make sense....

I have FCPHD on a G4 and if I get the HVX and a firestore storage unit, all I would need at that point is a terrabyte storage unit (any favorites?) and I suppose a SCSI connector and I could be doing some HD editing, correct?

Currently I have a Lacie drive which I used to edit a DVX100A shot feature but have never tried to do any uncompressed or real hi res work before.

 

[braw]

My POV really is when you shoot miniDV its archived on the spot as long as you keep your tape. When dealing with P2 you have to have a work flow that is mindful of your raw footage.

This has been hashed out a bunch on this forum. MniDV and HDV are very friendly to the footage materbat or. Shoot everything god gave you and hash it out later. P2 forces a more film like workflow, mostly because of technological restrictions. I'm sure if Panasonic, or any other company could give us many hours of solid state recording for pennies they would. Eventually they will. This is very new ground...especially for high def.

IMHO, if you have a 10 or 20 to1 shooting ratio, you haven't worked out your script very well, or you probably didn't need that last tab of acid to bring your stunning vision to life. If you're doing a documentary or something that requires a bunch of footage, then HDV should do just fine for you. My guess is that you won't be shooting much action or need much freedom for post work.

 

[OliverM]

2,5 TB in raid from LACIE will set you back about 3400$.
And remember the prices are going down fast. And the space DVCProHD takes up stays the same...

 

[WestEnd]

IMHO, if you have a 10 or 20 to1 shooting ratio, you haven't worked out your script very well, or you probably didn't need that last tab of acid to bring your stunning vision to life.

I couldn't agree more. In film school we worked with 16mm and being a broke student film was at a premium, so you had to learn to shoot 1:1 I couldn't begin to imagine needing 10:1 After shooting a 90 minute feature on HD I'm guessing I would only have 3 to 5 hours of footage. After editing I might have my 90 minute feature plus maybe 30 to 60 minutes of extras I might keep. All easily archived on one Hard Drive.

 

[wigby]

film school look

film school look

I couldn't agree more. In film school we worked with 16mm and being a broke student film was at a premium, so you had to learn to shoot 1:1 I couldn't begin to imagine needing 10:1 After shooting a 90 minute feature on HD I'm guessing I would only have 3 to 5 hours of footage. After editing I might have my 90 minute feature plus maybe 30 to 60 minutes of extras I might keep. All easily archived on one Hard Drive.

don't want to get too far off topic but...

the reason why all student films look the way they do is because all students shoot 1:1. i'm all for low ratios. i shot my 100 minute feature on super 16 at around 5:1. it looked good but you can't hide "2 takes and moving on" especially when compared to million dollar budgets with "20 takes until we get it perfect." when you compare film to video, setups require the same crew, planning and time. why not shoot a few extra takes in an effort to achieve perfection? it's the cheap hd video media costs (p2 isn't that expensive compared to 35mm) that filmmakers have dreamed of for years. now that we have, why not exploit it?

 

[WestEnd]

don't want to get too far off topic but...

the reason why all student films look the way they do is because all students shoot 1:1. i'm all for low ratios. i shot my 100 minute feature on super 16 at around 5:1. it looked good but you can't hide "2 takes and moving on" especially when compared to million dollar budgets with "20 takes until we get it perfect." when you compare film to video, setups require the same crew, planning and time. why not shoot a few extra takes in an effort to achieve perfection? it's the cheap hd video media costs (p2 isn't that expensive compared to 35mm) that filmmakers have dreamed of for years. now that we have, why not exploit it?

I agree, and I will shoot several takes. I just learned to rehearse over and over to make it perfect before rolling the camera. I plan to use a firestore so I will have plenty of space for lots of takes.

 

[Tibby]

This has been hashed out a bunch on this forum. MniDV and HDV are very friendly to the footage materbat or. Shoot everything god gave you and hash it out later. P2 forces a more film like workflow, mostly because of technological restrictions.

What exactly is a materbat? :undecided

I agree that here are to many wannabe filmakers out there that shoot everything in plain sight and hope they have a movie in the editing room. Maybe the HVX200 will be a great abject lesson for them.

But, what people seem to forget (or maybe overlook) in their zealotry towards anything panasonic on these boards is this:

Lets say you have no hard drive crashes and you are able to edit and output your piece. Okay great, but what are you gonna do keep buying raid array hard drives? Not likely! You'll erase that footage. Gone. The footage is gone completely. No getting it back. Film workflow maybe, but you can still archive film at the end of the day.

There is no practical method for archiving DVCPROHD material, and it doesn't look like there will be for some time. At least not for my budgetary constraints.

 

[braw]

the reason why all student films look the way they do is because all students shoot 1:1. i'm all for low ratios. i shot my 100 minute feature on super 16 at around 5:1. it looked good but you can't hide "2 takes and moving on" especially when compared to million dollar budgets with "20 takes until we get it perfect."

Here's the major difference. You can in fact shoot 20 to 1 if you want. Just review the takes and immediately discard in camera. I am not trying to suggest you be a one take wonder. But why keep crap. At best it would come down to two or three takes that you haven't quite settled on. Just delete the ones you know don't work in camera before dumping, or scrap them right after you know it was a bad take. On tape, you kept it because you had cheap stock and sorted through it later. With film you need to be on, or have tons of money to burn. With P2, you're right in the middle.

And wigby, the real reson student films look the way they do is simply lack of experience. The whole process is to learn how to work in an established environment that requires discipline. Most film students do it because of ego, and that shines through.

 

[braw]

What exactly is a materbat?

Hitting the space bar too soon. Also assuming that this mistake would be picked up so no edit was necesarry. We all know what assuming gets you.

masterbator

But, what people seem to forget (or maybe overlook) in their zealotry towards anything panasonic on these boards is this:

Lets say you have no hard drive crashes and you are able to edit and output your piece. Okay great, but what are you gonna do keep buying raid array hard drives? Not likely! You'll erase that footage. Gone. The footage is gone completely. No getting it back. Film workflow maybe, but you can still archive film at the end of the day.

I again ask you to look at a DIY RAID 5 solution. If you're spending $5,995-$10,000 for a HD camera, and expect that to be the end of it, why are you bothering? Film making and videography have never been cheap, but recently have been made insanely more "affordable".

I'll try my point again. You really only need the RAID for safety in editing. Once edited, dump the final to tape, DVD, Hard Drive, or whatever solution exists at the time. There are options, it's just different than what most are used to. For me, it's a workflow that seems to force a tidy, better management of footage. To each his own.

Shove it for calling anyone on this board a Panasonic "butt boy". I only gravitated to this board because it was all about a camera that I was massively impressed with. Plus, there are so many insanely informed and massively intelligent folks here. Watch what happens if Panasonic were to drop the ball.

There is no practical method for archiving DVCPROHD material...

That's just ignant.