FS7: Speed up my offload workflow

egproductions

egproductions

Veteran
I am currently shooting on Lexar 64GB XQD cards (1333x). While they suite my needs 90% of the time, occasionally I have jobs where I will shoot all day and then have to do an all night edit. In those situations, waiting to offload 8 cards really sets me back and can take 2-3 hours. I want to shorten this time as much as possible.Here is what the workflow looks like:

  • Shoot on 8-10 64GB 1333x XQD cards
  • Transfer one at a time using the Sony card reader plugged into an anker USB 3.0 HUB
  • Transfer using windows explorer or total commander
  • Transfer speeds to my RAID 10 is roughly 110MB/s (RAID 10 is also connected via USB 3.0 directly to the motherboard)
  • Import footage into Premiere Pro and begin editing


I actually have two Sony card readers, however, when I transfer using two at a time, each card transfers at around 30MB/s slowing down more than half. This happens whether both are plugged into the USB HUB or when one is via the HUB and one is directly to the motherboard (which leads me to believe the HUB isn't the thing slowing it down)

Should I be looking at getting a faster RAID setup/drive? (The RAID is a Sansdigital TR4UTBPN 4 drive Hardware Raid using 4 6TB WD red drives in a RAID 10 Configuration)
Would buying faster XQD cards help me at all?
Are there any programs that would help transfer faster than windows explorer?

Thank you!
 
First of all you need this usb/thunderbolt reader (if you have thunderbolt), it allows you to read your SD cards at full speed at the same time (max 4)

Lexar Usb/thundelbolt hub

if you only have usb 3 search for the USB3.0 only version.

I would suggest also using a scratch WIP ssd (samsung evo 500gb should do) so you have enough bandwidth (more or less 450/MB/s real world usage).

I use this setup and here's my workflow:

- Put all Sd Cards in the sandisk hub all at the same time (I use several 64gb sandisk 90MB/s)
- Use Hedge for mac to offload all the contents (excellent program, it does a checksum verification and unmounts the card when finished) and meanwhile organize premiere pro project
- Those contents go to a WorkInProgress 512SSD (total offoad time for 4x64gb full cards is 12 minutes)
- Start logging an editing while Carbon Copy is making the backup from the ssd to my RAID 5 station
- When project is finished or ready for approval i do all the backups and clear the WIP ssd so it's ready for the next project

Total time from inserting the first card to viewing the first scene in premiere is under 20/25 minutes.

There's no risk of data lost because at every step i have at least 2 copy of all media and projects.
It's a little expensive (more or less 500$ for the fastest configuration) but the speed i gained is tremedous, and it also allowed me to have less expensive/fast SDs around.
 
Last edited:
Is the speed of the USB 3.0 the determinate of transfer speed?

Or, let me put it this way- I am looking into purchasing a pre-owned macbook to use for transferring media in the field. Will my transfer speed be effected by the speed of the laptop's processor? Or is the speed of the USB always going to be the determinate for the transfer speed and it doesn't matter how fast the laptop's processor? I was under the impression that the 13" Macbook Pro's have limitations that make them a poor choice over the more expensive 15" macbook pro's. But someone who uses a 13" macbook pro said that this is nonsense and a 13" Macbook pro can transfer media ( from XQD card to portable drive ) every bit as fast as a 15". If I am purchasing a laptop solely for transferring footage from XQD cards to a client's travel drive, is there any reason to get a 15" over a 13"?
 
I don't think so, JP. I bought the 13 inch for precisely the same reason you mention (check out apples refurbished store, i got a basically new computer for 900 dollars). I got a model with an SSD so the computer boots instantly, and has thunderbolt to connect my main edit drive, two USB 3.0 for field drives and card readers, and an HDMI port which has proven handy a few times since having it as well. I can't imagine the 15inch would transfer footage any faster - but having the SSD with instant boot time has been a great time save for sure.
 
I can't speak about how much the machine will matter , but if you get on a Mac "Hedge for Mac" is blazingly fast at doing 2 simultaneous backups . Also one thing I learned from their support is that a powered drive will generally go faster than a bus powered drive. I can download 1 128G XQD card in 15-18 min to 2 drives using a USB 3.0 or thunderbolt HUB on 2012 Macbook pro with 2 USB 3 ports.
Paul at Hedge support is very good at giving advice on how to speed up backups and they have a very god FAQ on their website.
 
would second that.. I had used SPP for many years.. and its a fine bit of software.. but I dont need any of the amazing complex folder nesting etc stuff it can do.. and hedge is faster for straight card to HDD .. basic stuff.. does give you more chance of dinner at he end of a long day ! and the app for remote notification is nice.. SPP changed to gmail only notification for some reason .. a pain ..
Powered drives are faster .. but bigger and you need a room with the extra power point.. which can be difficult in alot of countries..
 
I,d rather eat my lunch.. :).. I prefer to do the down loading at the end of the day ,if there is no DIT and its down to me.. in a calm way with everything laid out on a desk at a hotel or home.. mistakes are 99% user error.. not in the back of a car..suddenly you have to move.. your not there to make sure some clown kicks out the AC.. is the card down loaded or not.. all the thing s that WILL go wrong .. because they can.. same reason I never let the director "help" me .. one person job.. calm atmosphere.. no rush.. methodical pattern.. down loading as you shoot.. nuts and guaranteed to bite you in the arse one day..
 
Donny 123 said:
would second that.. I had used SPP for many years.. and its a fine bit of software.. but I dont need any of the amazing complex folder nesting etc stuff it can do.. and hedge is faster for straight card to HDD .. basic stuff.. does give you more chance of dinner at he end of a long day ! and the app for remote notification is nice.. SPP changed to gmail only notification for some reason .. a pain ..
Powered drives are faster .. but bigger and you need a room with the extra power point.. which can be difficult in alot of countries..
Click to expand...

Is using Hedge faster than just drag & drop sans any application?


My footage shot in ProRes is transferring roughly in real time. That is, an hour of ProRes takes an hour to transfer, sans any transfer management program. Just straight drag and drop.


"Lunch"? What is this "lunch" of which you speak? Oh, you mean the period where you stuff down 1/3rd of a Subway turkey sandwich handed to you by an apologetic client who reminds you that "we" are way behind schedule and this next interview is the top guy so it really needs to look nice and he is going to be here any moment.
 
Indeed. Lunch, competent assistant/Dit, a break before going into an edit after a gruelling shoot day - all things that a production does not require.. and people come on dvx looking for a solution for quick card dumping to help the production not require those things!

I'm sorry op I can't really help. For really fast turnaround I have recorded to sdd with a shuttle and thrown that in my computer for a fast edit.. But that was only viable for quick shoots.
 
JPNola said:
Is using Hedge faster than just drag & drop sans any application?


My footage shot in ProRes is transferring roughly in real time. That is, an hour of ProRes takes an hour to transfer, sans any transfer management program. Just straight drag and drop.


"Lunch"? What is this "lunch" of which you speak? Oh, you mean the period where you stuff down 1/3rd of a Subway turkey sandwich handed to you by an apologetic client who reminds you that "we" are way behind schedule and this next interview is the top guy so it really needs to look nice and he is going to be here any moment.
Click to expand...

I believe hedge say their down load is only slightly slower than drag and drop..I haven't checked that as I,ll only drag and drop as totally knackered pissed off, plane to catch alternative.. and even then feel a bit guilty ..but it is alot faster than SPP..
Your down load should be faster than real time though ..esp if only drag and drop .. thats a bit odd I think.. with SxS full 128GB 130 mins XAVC HD with SPP down load to 2 X drives was about 45 mins.. with Hedge seems around 20-25 mins.. 2012 Air mac USB3 drives TB Sonnet reader..
 
Donny 123 said:
I believe hedge say their down load is only slightly slower than drag and drop..I haven't checked that as I,ll only drag and drop as totally knackered pissed off, plane to catch alternative.. and even then feel a bit guilty ..but it is alot faster than SPP..
Your down load should be faster than real time though ..esp if only drag and drop .. thats a bit odd I think.. with SxS full 128GB 130 mins XAVC HD with SPP down load to 2 X drives was about 45 mins.. with Hedge seems around 20-25 mins.. 2012 Air mac USB3 drives TB Sonnet reader..
Click to expand...

The hedge forum recommends using ac powered external drives and not bus-powered external drives. And a forum post says they are roughly the same price. What are some of these ac powered external drives that are the same price ( $59-79 ) as the bus powered external drives?

I am tempted to start using the ac powered drives since they make for more speedy transfers, but I doubt that my clients will much like drives that require ac power. Many Field Producers view / log the footage while on their return flight home. The ac power requirement would be a pain for them. The standard in our industry for years has been the Lacie drives. The ones with the orange plastic bumper thingy. I have stacks of those that were used to back up footage from EPK's.
 
Unfortunately its very hard to find AC powered drives at retail stores like Best Buy don't sell AC powered small 2.5" drives and that's where my clients usually buy there's or where I need to go at the last minute because they forgot to bring drives. If you time and foresight though OWC sells very nice small ext drives that are either bus powered or AC. The best of both worlds.

You need to request the AC adapter though for another $10
 
Last edited:
JPNola said:
The hedge forum recommends using ac powered external drives and not bus-powered external drives. And a forum post says they are roughly the same price. What are some of these ac powered external drives that are the same price ( $59-79 ) as the bus powered external drives?

I am tempted to start using the ac powered drives since they make for more speedy transfers, but I doubt that my clients will much like drives that require ac power. Many Field Producers view / log the footage while on their return flight home. The ac power requirement would be a pain for them. The standard in our industry for years has been the Lacie drives. The ones with the orange plastic bumper thingy. I have stacks of those that were used to back up footage from EPK's.
Click to expand...

Yes AC powered HDD are faster.. thats not really news.. but they used to be alot bigger.. maybe not now.. but as you say it means they (Dir/pro) need AC to have a look too and its just another cable.. and I have been in places.. Korea comes to mind, where there was literally 1 power point in the whole room... or 2 one at each end of the room.. AC drives became a real pain.. I always like to have my computer on mains. just in case !!
SSD drives are faster too I believe .. but exp and smaller GB,s..
But your down loads should be alot faster than real time.. esp as you arnt using any software to slow it down.. is there some driver you dont have installed maybe..
 
LennyLevy said:
Donny,
That's why I suggested the OWC portable drives because they are small and can be bus powered or AC. Only available mail order though.

https://eshop.macsales.com/shop/external-storage

"On the go" or "Mercury Elite Mini". Not sure what the difference is besides the housing & price. They use different chipsets.

Im quite happy with my down load times myself.. Hedge has helped there.. I think more than any drives or AC power.. For my work I cant really mail order drives, a recognized manufacturer is expected.. and often they are supplied by the prod co.. who wont get special ones for me I doubt.. in a freelance DP world being specific about drives doesn't work..or building my own.. .. AC powered HUB will help for sure.. if you use a HUB..roll on SSD drives .. that will happen sooner or later at larger GB,s..

I think the OP has some other bigger problem if by even drag and drop his down loads are taking the same as recording time.. thats like copying tapes !!.. some missing driver I think.. but an easy way to speed things up does seem to be Hedge.. as compared to SPP anyway.. if you dont need the more advanced features of SPP..thumbnail s on reports etc.. which is still a very good software, I still have on my computer if needed.. but simple copy with verification Hedge is faster..in my experience anyway..
Click to expand...
 
Absolutely transfer to SSD drives if you're trying to minimize your Xfer times!
Always look for your speed bottleneck in any situation.
If clients are bringing drives, they probably won't want to pony up for ssd's - though at this point even Costco sells a reasonable 500 GB USB3 ssd drive for about $125. I seem to get about 10GB/min speed using the USB3 ports on a 2015 13" Macbook Pro - and that's not necessarily with the fastest cards.
When I know there's a lot to transfer (or if the client is bringing their own inexpensive portable HDD drives) I often bring a second laptop computer.
And of course always bring spare drives for when the client forgets theirs - or doesn't bring enough data space. (And always charge a daily rental for drives they 'borrow' - or you won't get them back for weeks!)
 
You must log in or register to reply here.
Back
Top