I have been using my FS700 and Convergent Design 7Q in RAW>ProRes mode for a long time with great results. I am now experimenting with the next step, shooting Cinema DNG. I am curious as to how others are dealing with file transfer. I have bought a OWC Thunderbolt bare drive dock that allows me to plug the CD SSD card into one eSata interface and any 3.5" HDD into the other. The maximum transfer speeds I'm seeing are 60-80 MBS, not bad, but still cumbersome for 4K Raw footage. I'm assuming the transfer rate is limited by the read/write speed of the drives. Has anyone seen a better way to do this? I'm thinking maybe a TBolt RAID 0 drive with 7200 RPM disks?
Raw File Transfer Strategy
- Thread starter ric8138
- Start date
Shooting 4K PRORES, I use up 1TB for every 3 hours. When I get home/hotel/motel, I connect each SSD to each NAS, and tell the NAS to copy from USB/NAS to it internal HD - by next morning, all are done. Then when the filming is done, I have the time to download to main computer and convert
Thanks, but in 4K DNG Raw, I'm looking at about 32 min of footage per Terabite, so transfer during the shoot is a necessity. I want to slam an Ssd into a slot and walk away and have that SSD ready to format and reload ASAP. Doing it onto a RAID 1 would guaranty 2 copies. Thunderbolt can carry 10G/sec so I could also offload to 2 devices at the same time. I tried the same dock to daisy chained 1T Lacie external Thunderbolt drive and only got about 60 MBS on that chain. I see some hi performance SATA drives that claim to be capable of 6g/sec. Could I expect something in that realm in the OWC dock with the SSD in one slot a high performance drive in the other?
I would not use a direct file copy without some form of checksum verification for DNG raw. You can get an occasional corrupted frame or two that way. I use Raw4Pro to offload camera drives via a PC, gives copy verification and writes video proxies in a choice of formats at the same time. H.264 proxy, Prores HQ422 or 444, DNXHD at a similar choice, Cineform 10 bit 4:2:2 or 10 bit 4:4:4 too. Never lost a frame that way.
Cyril-from-France
Member
In my case, hard drives are just for archiving or to secure. I only work with SSDs. Thunderbolt is useless with 7200 RPM disks.
I guess this is the point, I am looking for a way to inexpensively transfer my files, rapidly, from my shooting SSD's to lower cost HDD's. Thunderbolt as the bus would, in theory allow a simultaneous transfer from the shooting SSD to two Thunderbolt connected HDD's at the highest data rate possible to write to the HDD. Test/review sites show real world transfer rates of 180MBS for better HDD's, which, although not great, is much better than what I am seeing (60-80 MBS).
doubledark
Well-known member
Can I ask a noob question; what is the point of doing checksum verification?
Another words, lets say the check fails; then what?
If the check fails that would indicate an error in the transfer. In my case that the contents of the shooting SSD was not perfectly copied to the storage media. Hopefully this would stop you from reusing the SSD, finding the error, and performing an accurate copy. In my case I always record to the camera's internal system as well, and for most clients, they really can't tell the difference between AVCHD and say ProRes. When shooting long form narrative though, where grading is important, there will be a major difference in visual quality.
cpc
Well-known member
Assuming your bottleneck is the output HDD (and not the SSD), you can double the transfer speed by offloading with slimraw (/shameless self promotion). It will losslessly shrink the files on the fly, decreasing the necessary write-out throughput. Also, checksums, simultaneous offload to two drives and stuff like that.
doubledark
Well-known member
So the error could be with either the shooting SSD or the back up SSD. I imagine the more data held on the camera side the less practical it becomes to re-shoot if you discover an issue.
David W. Jones
Veteran
You could put together an inexpensive SSD Thunderbolt raid with components from OWC which would be very fast.
OWC has SSD RAIDs preconfigured. I guess I'm trying to figure out where the bottleneck is now, and where the sweet spot is for reliability, convenience and cost. The Thunderbolt dock that I bought from them appears to transfer data up the Tbolt to the laptop then back to the receiving drive, rather that just across the TBolt connection. It is a TBolt2 device w 2 TBolt connections, so theoretically, I should be able to approach the R/W speed of the CD shooting media (256G SSD) which looks to be 500MBS. That would be a pretty efficient transfer rate for on set backup. I would welcome comments from any users of SlimRaw and ShotPutPro as to their usefulness and speed of transfer qualities.
OWC sells SSDs with SandForce controllers that use compression to achieve their write speed. The relevant number for you is Incompressible Write Speed.
https://eshop.macsales.com/shop/SSD/OWC/Mercury_6G/
If you are using Odyssey7Q to record, the Samsung 850 EVO 1TB is pretty close to the cheapest you can get on cost per GB for SSDs. They are certified for all recording modes in the O7Q. The fastest 'backup' would be to just buy more of those. Dual record is supposedly coming to more formats.
Using XXHash64 instead of MD5 verification in ShotPut speeds up file verification a lot for me, specifically for formats that use single file for each frame. For large ProRes files it's not that different from MD5 verification. Your mileage may vary depending on computer setup.
You should also check max speed of the various Thunderbolt Docks you have. My Seagate Thunderbolt to SATA adapters max out at 370, which is quite normal for a lot of Thunderbolt docks, while f.ex. Akitio SATA2Go will read up to 530, apparently. There are also USB 3.0 SATA adapters that are faster than 370.
Thanks for the info! That's pretty fast!
One more note: I always get a slowdown when I deal with formats that store as single file per frame instead of a single file per take, like f.ex. .dpx or .rmf (Canon Cinema RAW) or .ari (Arri RAW). I haven't dealt much with Adobe Cinema DNG yet, but it should be similar in terms of slowdown in data copying speed because there's lots of small files instead of a few big files. It's not just the verification process that takes longer (because there's so many more files that need to be verified). F.ex. when I copy any of the formats mentioned above to a very fast SSD I am typically quite a bit below the max of the Seagate Thunderbolt adapter's max at 370, even though both SSDs are capable of going above 500 read & write.
Worth noting is that ARRI has changed their wrapper for .ari in the Alexa Mini to a new .MXF wrapper so it's one big file (apparently, I haven't tried yet). That new wrapper should allow for higher frame rates to the CFast cards and quicker offload.
And before anybody asks: Large file wrapper is not necessarily a good idea for Convergent Design's Odyssey7Q since they utilize the benefit of single file per frame format in a very clever way.
I have ordered a Samsung 850 EVO 1TB and expect it in today. I have 4 256G CD SSD's in my kit now. If the transfer speed is up there on the Samsung and it records without issue on the 7Q, that would appear to be the best answer for a day's shooting on set, then transfer to a conventional TBolt RAID for edit, and storage. Then bounce finished, managed project back to cheap HDD's in the dock for archive. Thanks all for your input.
Programs like Raw4Pro use checksum to verify and correct transfer errors. Mobile media readers are especially prone to transfer errors. It is a matter of low probability, but when you are transfering the massive amounts of data raw shooting produces, a dropped or corrupted frame here or there can be surprisingly common. Usually repeating the transfer of a corrupted file corrects it. Raw video consist of stacks of individual frame files in a separate folder for each shot.
80MB/sec is about the max I get transferring to single 7200rpm drives via USB3. RAID 0 definitely helps. I use a hot swap drive bay and 2TB NAS drives for media. Less than $100 each these days. Much cheaper than say HDCAM SR tape.
Last edited:
If you had the SSDs in separate readers connected to separate busses you could get almost twice that speed.
