Long Term Recording Direct To Disc

I did a three camera shoot a few months ago involving long (3 hour) interviews each day for several weeks, shot 3 camera, all being HVXs. All 3 cameras were recorded direct to disk in FCP, using mirror arrays for redundancy. (just 2 drives per computer). One of these arrays was USB.

I'll get to your specific USB question later on, but first maybe it's useful to paint the full picture, as it sounds like it may be useful to what you're attempting:

All shooting was 720p 24p, and pulldown was done in FCP during capture so I would have the file-size savings during capture (as if I'd been doing 24pn in terms of the resulting file size and format after capture.)

We had 50 and 75 foot runs to the capture station in our video village, so I used repeaters and special extra-high quality 10 meter FW400 cable with Quad shielding (S-400 specification) for maximum speed without data loss, that had been recommended to me. They're sold by http://www.unibrain.com/ and made in Italy. Do seem to be well made. Mick there seemed to know his stuff and took my order.

10 meter between repeaters is longer than the firewire spec (by twice) but we didn't have a significant problem with it for the most part. As I understand it, with high quality cable, the 5 meter spec limit pertains specifically to runs where power is being drawn from the FW ports, which is not the case during capture with HVXs.

Cables: http://www.unibrain.com/Products/p1394/cables_a.htm
Repeaters: http://www.unibrain.com/Products/p1394/FR400.htm

Coated all FW connections with stabilant 22. Had a few glitches some days, but nothing major. We placed repeaters in tupperware containers, taped up with strain relief, so no worries about foot traffic causing problems.

As mentioned elsewhere in this thread, we stopped recording when possible every twenty minutes or so for safety. It only took a few seconds to stop and restart all 3 computers, and didn't seem a problem.

As per the HDPro spec, The FW signal feeding INTO the computers was 100mbs, but the signal going out to the capture drives was half that, approximately. due to the pulldown removal in FCP. With two tracks of audio, FCP reported the data rate as about 6mb/s

Two capture computers were Powerbooks, with the camera coming in FW400 and going out via PCMCIA cardbus FW800 into the drives.

One was a new 17" iMac Intel. To keep separate buses on that unit, since there's only one FW bus possible there, I used USB 2 out to the drives. Used Mirrored Wiebetech drives (Traydock style and Silversata) which employ SATA drives but have choice of FW800 or USB 2 i/o. Tested using Aja's utility, and got 15mb/sec write speed on these drives. Less than half the speed of my FW 800 mirrors, but still twice the speed required to do the job.

We did one modification to the HVX camera A, an external mechanical zoom knob adaptation to allow undetectably slow creep zooms during the interviews.

At some point I'll do a more thorough writeup on all this for DVX user if there is interest. Right now I'm getting ready for our next location shoot on this project, so it may be a while though. There are some pictures on www.westwoodcreek.com on the more info page (via the purple link right above the group photo).
 
Forgot to mention -- one way I determined that the FW repeater situation seemd up to snuff was, pre-shoot, did 75 minute 720p captures using the full repeater rig for each camera, and also added a FW coupler between the chain and the computer (a non-repeater coupler not normally designed for this purpose) and a single additional 6' FW cable after this coupler. This was done to lower the intregrity of the chain, creating a worst-case scenario. We figured that if we didn't have dropped frames then, we'd be safe once the coupler was out of the game.
 
dvxStephen,
thank you so much, I REALLY have been waiting for an answer like that...
Do you know if your capturing setup would be possible to do with two separate USB/FW Discs?
I´m asking this, because I´m not sure about the editing and archiving workflow.
I was thinking, if I had two separate drives, I simply can put one in the shelves after filming and save the time for the backup of the external raid...
At least thats what I guess you´re doing: capturing to the external raids on location in mirror mode, than later do a final backup on one single disc for the archive, am I right?
I will do this on a more or less regular basis, we´ll have to do a lot of encoding after editing, so baking up another time also becomes a time issue...
 
Two of my mirrored capture drive pairs are made up of single G-Tech 500 G-Drives. No fan. Nice design. Stay pretty cool. And quiet. Mirror so far via Apple's Disk Utility.

Downside: No error status report. If one drive drops out, you have to be watching the drive activity lights (or go back to Disk Utility) for a status check. Upside: Simple and free, and usually works. But if you allow one drive of the pair to start alone and be seen by the Mac finder at ANY time, the mirror is out of sync. It drops out of mirror mode, and depending on how you set it up in Apple's utility, it may initiate an auto-rebuild- even if the data is fine! that takes 3 hours w/500 gb drives.

In a pinch you could still record to one of the drives, though... I dealt with this by being very careful with how I turned things on and off, and also having a spare mirror drive pair on hand.

Softraid makes a software mirror solution that has the 3 hour auto rebuild problem also, but at least reports errors, and has some other advantages. Not sure what disadvantages other than cost. I've got it but haven't tested it yet.

To answer what I think is your main question, I guess you could split up the mirror afterwards and use one drive for editing and one for backup.

In my case, I erased the mirrors when full (after offloading the data of course) and started over. Every day or three, after the day's shoot, I'd offload the data from each mirror pair to three stand-alone SATA drives, which is where the data lives once recorded. One for editing. One for client-kept, off-site storage as a safety. And one for my primary backup, also off site, as a safety. Three is the minimum number of drives recommended by IT people for critical data.

Stephen
 
hey Stephen,
so how often do the mirrored G-Tech drives fail you?

And in which way are the wiebetech drives better? Because they have a display and inform you about drop outs?

What do you mean by one of the drives starting alone? I didn´t know they could when set up as a raid.

If you back up to three SATA drives, how long does this take for let´s say 300GB?
Do you have to copy on after the other or can you simultaneously offload to 3 drives, provided that the computer has enough ports...?

Last, do you charge your client for every drive?

Thank you so much for the insight, I appreciate that so much!

EDIT: I forgot to ask sth really important, does the recording time direct to disc differ from P2 recording times?
I did all my calculations on the needed disc capacity by just multplying my P2 cards recording time,
so I suppose I´ll need 34GB for 1 hour off DVCproHD@720p...?
 
Last edited:
My HVX manual clearly states that it is not recommended to out put via USB port to MACs (F/W is fine). The opposite is stated for PCs
 
Shooter said:
My HVX manual clearly states that it is not recommended to out put via USB port to MACs (F/W is fine). The opposite is stated for PCs
Click to expand...
Thanks shooter, I read that too. USB would only be an option for connecting external drives in my case, not the HVX.

Regards,

Spart
 
Spartacus said:
EDIT: I forgot to ask sth really important, does the recording time direct to disc differ from P2 recording times?
I did all my calculations on the needed disc capacity by just multplying my P2 cards recording time,
so I suppose I´ll need 34GB for 1 hour off DVCproHD@720p...?
Click to expand...


Well, yes and no. Output theough firewire is always 100Mbps ("over 60" rather than "native" mode). However, FCP can remove pulldown on the fly, so that the storage requirements for SOME things are the same as "native" mode on a card. I can assure you that 720p30 and 720p24, 480p30 and 480p24 can work this way. I'm pretty sure variable framerates will take the full space of the "base frame rate", and there seem to be some issues withFCPs live pulldown-removal 1080p modes that may or may not have been solved (lots of discussion on that here).
 
"so how often do the mirrored G-Tech drives fail you?"

Havent' had a failure yet. Every once in a while FCP complains about a dropped frame or whatever, and we'd stop all cameras and restart. Some days we'd go for 3 hours without a glitch. Some days we had 2 or 3 glitches, or even 5. Seemed sort of random. We traced one string of these to static electricity, ground issues, or something like that when our HVX on our metal jib crane arm contacted our lighting effects motor platform's metal frame, since the jib worked in that area frequently with tight quarters (see the slides on my web site, referenced above in my first post on this page, for the visual of this).

Stabilant 22 seemed to help on FW connections, as mentioned before. Stable power seemed to help. Not having a full moon seemed to help. So did chocolate. (For the operators that is -- I recommend Scharffenberger 70% -- in fact, I became a dealer so I could pass it out freely to crew -- but that's another tale!)


"And in which way are the wiebetech drives better? Because they have a display and inform you about drop outs?"

The Wiebetech SilverSata line has a 2-bay hardware style RAID with basic built in diagnostics, that's nicely designed for Raid O or 1. Fan cooled, which is a good idea with two drives in a box, and the fan speed can even be changed via software they supply -- if you know to look for it; not in the manual. (And windows only via a serial cable they supply, far as adusting fan speed goes.)

In my case, I abandoned using these boxes after finding they don't work reliably with all Hitachi drives -- firmware conflicts, which I ran into. I've standardized on Hitachi, so I couldn't use this box unfortunately. In hindsight I might have bought a couple drives from other than Hitachi that they certify as compatible and stayed with this box, as it is a good one.

Advantages - only one plug on back, so you can't accidentally unplug one drive and 'break' the mirror. Fan cooling. Nice tray design. Good price. eSata or USB i/o.

Disadvantages (besides the Hitachi question): No Firewire out. eSata Cardbus cards exist, but I haven't tested 'em on Macbook Pro.

Sata cards were a bit slow on my Powerbook due, evidently, to some i/o problems with the PCMCIA slot's internal bus i/o on even recent powerbooks, or so FW Depot's tech people claimed. Anyway, two-drive SATA write speeds weren't what you'd expect, and a pair of SATAs didn't seem to mirror well, via PCMCIA cardbus -- but it's been months since I tested this, and I never tested it with the Silversata II RAID. (I tested with a pair of Firmtek Seritek SATA drives at the time, using software mirror. Works fine for one-drive-at-a-time copies, but not for both drives doing i/o at once. I'm rusty on the details as it's been months. Still use the Seritek SATA housings for backups, and they work fine for that. And also editing back in the edit suite.)

So, for one of my mirrors, I'm going with the Wiebetech SilverSata IIFH, which has Firewire 800 and USB. No hardware raid, but it's an all-in-one box -- Two sata drives inside in trays, and only one i/o cable needed. Which brings me to...

"What do you mean by one of the drives starting alone? I didn´t know they could when set up as a raid."

Let's say you have 2 G-Tech G-Drives sitting on the table, working in a software RAID mirror. You accidentally pull the plug on one, the firewire cable falls out, etc. If it's not connected to your Mac, no problem. BUt if it IS, the Finder writes different date markers to the drive that's still up, and you have instant-out-of sync situation on your hands, as far as the Mirror is concerned. So the Mirror will report being broken, even though no actual data is out of sync besides file dates unless you capture in this state. The mirror will insist on rebuilding the second drive, sector by sector, which takes 3 hours with 500 gb drives -- or simply stop working as a mirror and not even notify you, if you're using Apple's mirror option built into their disk utility software and didn't specify auto-rebuild when setting up the mirror. You can also initiate a mirror rebuild with apple's software in this case, but you have to fire it up to do so.

Incidentally, this isn't QUITE as bad as it sounds, as in my tests, the rebuilding mirror will still function as a recording mirror even during the rebuild process. The rebuild process slows it down some, but not enough to matter at 720p, far as my speed tests indicate. And if it's just a loose plug-rebuild, the data on the second drive's sectors appear to be left alone until each sector's re-written, so you still (sort of mostly) have a working backup! But there's the 3 hour thing, and it doesn't feel as solid to me as I'd like to have this happen and still film with all that going on!

"If you back up to three SATA drives, how long does this take for let´s say 300GB?"

Don't remember exactly, as I tend to do these at night, and walk away. Seems to be something like 1 hour per 100gb as I recall. Might be twice that. Depends too of course on all your particulars.

"Do you have to copy on after the other or can you simultaneously offload to 3 drives, provided that the computer has enough ports...?"

I do one at a time per laptop, being the cautious type. But since you'll presumably have more than one laptop? you can do more than one set at a time even so. I imagine you could do 2 per mac without a massive speed hit, but I haven't tested this. In any case, it helps to use different bus for in and out signals -- cardbus for one and native FW800 port for the other. And USB is WAY slower -- half FW400 at BEST.

"Last, do you charge your client for every drive?"

Depends on the client. In the current case, it made sense, as this is a large 3 year project, with dozens of 500 gb drives, and they agreed that it was essentially like film stock. They also were understanding, after I explained, regarding how the drives would be worth far less at the end of the project -- and they'd want to keep at least one data set anyway for archive purposes. I expect I'll buy a bunch back from them afterwards at a fair market rate at that point.

"EDIT: I forgot to ask sth really important, does the recording time direct to disc differ from P2 recording times?"

P2 recording times vary depending on the format. 720p/24pn charts give the same results I get after pulldown is removed doing 720p/24p. Since that's done on the fiy, the drives benefit from those savings. I get roughly 20 hours of video onto each 500gb drive with my current settings. DV would be about twice that at 24p, with pulldown removed.

I have not tested 1080 much yet. Waiting for FCP's update!
----------------------------------------------------------------------------
Shooter says,

"My HVX manual clearly states that it is not recommended to out put via USB port to MACs (F/W is fine). The opposite is stated for PCs"

If you read the above posts more carefully, I believe you'll find this is not the situation we've been discussing. We're connecting a drive's USB to a Mac's USB port, and the camera to that Mac's FW400 port, feeding Final Cut's live capture feature. The USB drive is only communicating with the Mac, functioning as Final Cut's scratch (capture) drive, and doesn't see or hear from the camera at all, beyond what FCP tells it to write, in real time, after FCP processes the data.

I hope you'll find this helpful. It may be 6 weeks or so before I can add to this thread, as I'm heading out for location soon.

Stephen
 
Stephen,
Thanks for your time and insight, though I got a lot of good advice here so far, yours has been one of the most detailed ones!
Good luck on the road!

Thanks a lot pal,

Spart
 
One last thought- If I was doing this over again, I'd try using the SilverSata II Hardware RAID via PCMCIA Sata card. Since it's a single I/O hardware raid, the I/O speeds via cardbus would probably be fine, even with the Powerbook line. If so, it might be an ideal setup.
 
But then I still would need to backup to different drives and later format the Raid to record on it again, right?
The client has decided to doing only one backup for budget reasions, if something happens to this drive, he knows the data is gone for sure and that it is his responsibilty.
Ahhhrg, those damn small budgets... ;o)
 
"But then I still would need to backup to different drives and later format the Raid to record on it again, right?"

Right.
 
I am reviving this thread, because I am considereing the SilverSATA and this is the only thread that came up in a search. Anyone else have anything else to add? Any more experiences?

PC laptop->USB2->SilverSATA 500gig RAID 1, 2 500 gig drives.

Shooting tapeless has me PARANOID, so RAID1

Anyway, if anyone else has some more experiences with this product, I'd love to hear them.
 
I've done this, but not using the particular drive you mention. I've recorded to an external USB2 7200 RPM drive using DV Rack 2.0 HD on a P4M 1.5GHz with no issues whatsoever.
 
Don´t know about the Silver Drives. We used the 320Gb G-Raid2s in the end, two raids daisy-chained via FW800 and the HVX on the FW400.
Only found out afterwards, that the FW bus will always allow only the slower speed (FW400) to be used, so basically I recorded my DV50 (Also tried 1080p...) simultaniously to two raids via the same FW400 bus - worked flawlessly for up to 4 hours.
One MBP and one PB G4.
Archieved on WD MyBookProsFW800.
I guess USB on a PC will be just as fine as FW on a Mac.
And DV50 an 720pn should have similar throughput...
 
thanks guys. I looked at G-RAID, but that only seems to do RAID0. I would like to avoid having to manually back-up footage as we go, so would like a RAID 1 solution. The other thing I like about the SilverSATA is that you can swap out drives and re-use the trays and RAID with new drives.

Even when daisey-chaining the G-RAID does not RAID over the FW.

I've done standard drives via USB2 with my laptop and 1080 and it all worked fine too, but my main concern is back-up. On an SD shoot if the laptop or drive craps, I have the tapes.
 
Mark Harris said:
Even when daisey-chaining the G-RAID does not RAID over the FW.
Click to expand...
Yes it does.
At least on a Mac.
We use two daisey chained G-Raids in Raid1...
Still do the archieving/final backup on the cheaper WD drives...
Hope that helps!
 
You must log in or register to reply here.
Back
Top