Weird file transfer issue- ruled everything else out

M. Gilden

M. Gilden

Canon DSLR Moderator
I want to share this in case anyone else has had something similar happen.

A while back I noticed while editing that I had a blank frame here and there in the video I was working on. I've had previews in Sony Vegas do weird things before, usually a restart of the computer fixes that right up. But sometimes I play back the original file in VLC and see the same frame has a weird green pixely area like it is missing part of the frame data (vegas is smart enough to blank out a frame rather than show corrupt data).

At the time, I was using a hacked GH2 and figured the camera's compressor couldn't keep up with the data rate or something like that. Weird things could happen from time to time with a hacked camera. When it happened on a different camera, I decided it was probably a sign of my SD card media starting to fail so I bought new cards.

But then I noticed it on a shoot recently with my A7s, using new SDXC cards. Same problem- occasionally a frame will be blank or have partial data, and this is both in the editor AND when playing the file straight off the drive. This was a real head scratcher. I put the original media back in the card reader, and the bad frames WEREN'T THERE. The original content was fine! So I recopied everything to my work drive, and it seemed to be fine. In all of these scenarios, the files were transferred simply by dragging the /PRIVATE folder off the SD card to my work drive.
So, if it wasn't the hacked camera, and it wasn't the SD media, maybe its my card reader! So I got another USB3 card reader and figured my problems are solved.

That is, until I noticed it on my data copied off my Atomos Ninja SSD! MOV files copied straight off the SSD using the official Atomos dock and copied using standard Windows drag-and-drop had corrupted frames on my drive. The originals played directly off the SSD did not. But the file copied correctly, there were no errors. For giggles, I downloaded an MD5 checksum application and tested the file giving me trouble against the original. Even though they were exactly the same size, the checksum didn't match. However other files from that session DID match. So something clearly has gotten messed up somewhere between the original media and my working drive.

Its not the camera, not the SD card, not the reader. Could it be the drive I've been using? That would be terrible, as I back up nightly but use that drive as my reference point, updating any files that changed. If my drive is corrupting the data, perhaps it has been corrupting all the data going to the backup as well! So I'm running a CHKDSK command on the drive now, and working off my backup in the mean time. So far no errors have been found, but its going to take hours if not days to scan this entire drive at this rate.

I'm very upset because all along I thought I had solved the problem each step of the way, only to find out it was something more. If the drive comes back clean from all the diagnostics, I don't even know what to think anymore! I feel like the first time I noticed this, I was using a different drive as well. However, I don't recall as it was a long time ago and I don't remember when I started using the one I have now. Has anyone ever dealt with something like this? I'm wondering if there is a tool that can copy data off my media while also doing an MD5 checksum or similar to verify the copy has worked. I'm having trouble trusting my workflow at this point.
 
It's a nightmare....ain't it?

I started using TERACOPY about a year ago for all file transfers because it will give you the checksum. Or data verification in other words.

Yeah, it takes a little longer... but losing frames is death.

Murder kill death. As in death by death.
 
DPStewart said:
It's a nightmare....ain't it?

I started using TERACOPY about a year ago for all file transfers because it will give you the checksum. Or data verification in other words.

Yeah, it takes a little longer... but losing frames is death.

Murder kill death. As in death by death.
Click to expand...

Seriously. Did you have an issue like this? I never assumed the Windows file transfer routine to be faulty. But now I'm wondering if my problems have been limited to just this drive, or if I can't trust it at all, ever. I don't know what to think now, this is madness. (not Sparta)
 
Sounds like the exact same thing that used to plague P2 users on Macs, the dreaded "glitch", which was rare but only fully resolvable by using a file verification copy process.

Some of the comments in that thread seemed to be implying that it was NTFS-related, others implied it was write-cache-related. I only read the first page, but those comments were from 3-4 years ago. Don't know if they're still asserting it as a current bug?

Unfortunately, yes, computers suck and it affects both major platforms. A verified file copy is your best defense against it.
 
Barry_Green said:
Sounds like the exact same thing that used to plague P2 users on Macs, the dreaded "glitch", which was rare but only fully resolvable by using a file verification copy process.

Some of the comments in that thread seemed to be implying that it was NTFS-related, others implied it was write-cache-related. I only read the first page, but those comments were from 3-4 years ago. Don't know if they're still asserting it as a current bug?

Unfortunately, yes, computers suck and it affects both major platforms. A verified file copy is your best defense against it.
Click to expand...

Thanks Barry-
Yes, after finding that thread, and many others on various other forums, I eventually found a more updated one here:
http://forum.videohelp.com/threads/367509-Files-corrupted-when-copying-to-external-usb3-hdd

This one at least has a lot more information and possible causes to try and experiment with, and is dated about a year ago (although the last post is me reviving it, dated today). Sounds like it could be Security Essentials causing it, or USB settings, or a combination of those things plus hardware. But they still aren't sure what causes it or definitively how to fix it.

I just can't believe this has been going on for years right under my nose. I noticed issues with my files now and then, but always blamed it on something else. I can't believe the problem was my operating system all along! Worse still is my open-source backup application (which I run nightly) uses windows file copy protocols under the hood, so if my working drive has occasionally corrupted data, I'd imagine my backup from that is even worse!
There aren't words to describe how I feel right now. You know that feeling in the pit of your stomach when you realize you've been lied to your whole life about something?

It seems like most users probably had this and didn't even notice. Only people working with large data sets that would notice a corrupted portion of data right away (like video people) would see it.

I like that Teracopy doesn't have this problem according to the users who tried it, and that it even has a checksum option to verify. That'll help for mission-critical transfers of data off storage media. But what about my incremental backups? I wish I could force it to use Teracopy, but that's just not how it works. I know there are backup applications that can verify data, but they also usually rely on the OS's copy protocols. So, yes- they would verify that the files didn't copy properly. Which means I'd have to keep running it until everything copies without errors. Are there any backup apps that use their own copy protocol (like Teracopy) so I can also just avoid this whole mess? Otherwise, I'm afraid I'm going to have a much bigger fish to fry!
 
Greg_E said:
Due to licensing requirements, we run McAfee at work.
Click to expand...

I don't think the security tool causes it. One guy said removing MSE seemed to improve his, but someone else who reformatted and tried it on a clean install of Win 7 x64 said it still had the problem. There was no virus scan even installed yet!

Others say that changing the USB connect settings to "high performance" improved, but may not have entirely eliminated, his corrupt copies. I personally don't think any of those are the culprit on their own, since it seems to be all over the place and no confirmed resolution for everyone. I think there's a real buffer/cache problem in the windows copy routine, and I think everyone has it... but it only rears its ugly head when a combination of USB drivers interfere with other things to create a perfect storm. Sometimes it could be the virus scan, sometimes the USB buffer, maybe it has to do with how much RAM is available... different things can cause it in different setups. So who knows?

If it was as simple as installing McAfee, I'd do that. But now that I've seen the bug and know my system does it, I don't think I could trust the Windows Copy function without some verification that the bug doesn't happen anymore.
 
Just wondering why we don't see it. I think I would still try the exFAT format and see if it still happens. If you need to use them on XP, there is a free exFAT driver from Microsoft. Several version of OSX ago implemented exFAT so you should be OK except for Linux machines.
 
Greg_E, in the technet thread linked above, some people were not able to recreate the problem on their end. That makes finding a cause or solution really dodgy. Some people could easily reproduce it, even on a fresh install (which sounds like hardware might be a big part of it). If your large copies check out OK (using something like an MD5 checksum to verify), then you're one of the lucky ones that aren't effected (yet?).

exFAT is a good idea! But I'm also not entirely sure it won't do it too - the people reporting the problem said it was on Win 7 x64 and NTSC formatted drives connected via USB. They confirmed that 32-bit windows didn't do it, but I don't recall them confirming that non-NTSC drives wouldn't. They might have just been reporting their experiences, which all happened to be NTSC format drives? I'm going to have to look into this more.

I don't know a whole lot about the difference between exFAT and NTSC, but I recall once reading that exFAT is less secure and has less error checking (irony). Then again, the drive from my Ninja 2 is already exFAT formatted. If its good enough for that, I'd guess exFAT is good enough for whatever else I'm doing. Bigger issue now is reformatting that whole sucker and copying everything back on, since you can't just switch format on the fly.
 
Just curious...have you tried going to a command prompt and using the XCOPY command? I know, I know -- it's an archaic way to go, but I'm just curious if you would experience the same problem.

There are two command line switches you can use with XCOPY that might be helpful.

/V Verifies the size of each new file.
/J Copies using unbuffered I/O. Recommended for very large files.

That last one in particular might be useful. Just a thought.
 
David W. Richardson said:
Just curious...have you tried going to a command prompt and using the XCOPY command? I know, I know -- it's an archaic way to go, but I'm just curious if you would experience the same problem.

There are two command line switches you can use with XCOPY that might be helpful.

/V Verifies the size of each new file.
/J Copies using unbuffered I/O. Recommended for very large files.

That last one in particular might be useful. Just a thought.
Click to expand...

I've heard that command prompt copy and Xcopy exhibit the same issue, as well as their third command line copy (the name escapes me at the moment). I don't know about /J, I'm not sure if anyone tried that. The verification by size won't matter, because the files ARE the same size in the end. Part of the data seems to get corrupted, but byte-for-byte, its there. It just got altered somewhere. I think this is why Windows doesn't report any errors in the process. The only way I noticed there was a problem was that frames were corrupted during my edit, and not on the original source material. An MD5 checksum confirms that some files copy correctly and others not, even tho all sizes match!
Teracopy seems to get around this bug and even has its own checksum verification. So I guess that's always an option, but my problem is that applications in windows that use the copy routine under the assumption that it is reliable. For example FreeFileSync mirrors my working folder to a folder on another drive as a daily backup. It uses Windows copy routines. Heck, I think some of my old logging tools for interseting metadata or stitching together spanned video files work by creating a temporary local file, then copy it to the destination when finished. That also uses Windows copy, and I can't force it to use TeraCopy or Xcopy /J or anything like that. It just isn't coded that way. So basically, this is a system-wide problem that some people either have, or don't. Apparently, I HAVE it. And no one knows what causes it exactly, or confirmed way to reliably fix it.
 
I can confirm it happened to a Gateway, Dell, and at least one home built machine according to the forums that people posted in. One said it started after a motherboard transplant, so perhaps something to do with a particular chipset. Some with the same USB controller chipset had no problem while others did, however. So we haven't narrowed it down yet. Maybe a combination of factors?
 
Some of them claimed eSATA too, so not just limited to USB. Since you seem to be big enough for this, I would suggest building a storage array. Anything bigger than 4 drives and if you can work it out, connected by iSCSi. FreeNAS is pretty easy to build and would provide fast enough connection to edit from if the pool is built correctly. I'd archive stuff onto "archival" Bluray disks from mDisc.

What's your normal bitrate on projects?
 
Greg_E said:
Some of them claimed eSATA too, so not just limited to USB. Since you seem to be big enough for this, I would suggest building a storage array. Anything bigger than 4 drives and if you can work it out, connected by iSCSi. FreeNAS is pretty easy to build and would provide fast enough connection to edit from if the pool is built correctly. I'd archive stuff onto "archival" Bluray disks from mDisc.

What's your normal bitrate on projects?
Click to expand...

That's an idea, but a whole fancy iSCSi setup will only help specifically my storage workflow. I'd like to know that I can somewhat trust a simple Windows file copy not to corrupt my data! There are times that I'm going to need to work off client-submitted material, or transfer back to an external drive to hand off to someone else... having a simple USB connection work is something I'll still need to tackle. I also like it that my current setup is simple enough that I can just take my USB working drive with me and edit on my laptop, then bring it back to my workstation later. I don't think that would work very well with a FreeNAS array. :/

However, it would be cool just to have for my own setup, so I'll look into that down the road. Meanwhile, I'm going to do some more testing and see if I can't figure out a way to make this more reliable.
 
Oh, and as far as eSATA doing it also... I'm not sure. One guy mentioned that, but I'm under the impression that eSATA works like an internal drive as far as file system access and cache/buffer for writing goes. Does the computer even know when something is connected via eSATA versus SATA? I know in my computer, I put in an eSATA bracket once, which was nothing more than cables that connected to my motherboards' available SATA ports, and moved them to something I could plug into the back. I haven't used eSATA in a while so I can't confirm if it does it now or ever did, but it looks like internal drives don't have this problem- only transfers to external HDs. And if Windows can't tell the difference between an externally connected eSATA and an internal SATA, why would one have the bug and the other not?
So I'm inclined to wonder if that eSATA user just had a drive that was starting to fail. That's the problem with this data- we don't don't have enough to make a scientific conclusion. Some people are effected, others aren't. I believe many people have it and aren't aware (like I wasn't until 48 hours ago), so I just don't have enough to go on without spending days doing my own experiments.
 
Some of the dedicated eSATA connections use a USB interface, so it depends on how you rig it. I think if you turn write caching on and make sure that the system requires safe removal you might be where it needs to be. Also disable any unconnected network ports (a known Intel issue) or make a loop through connector and plug it into the unused ports. Screwed with this last one for several years now, in theory the proper driver stops the extra noise from the unused port, but that doesn't always get installed. All my computers at work now have loop through plugs connected.
 
I use BackupChampion with Raw4Pro for raw file offloads. Now called Datamaster Pro. Data loss is an inherent risk of any data file transfer absent some kind of error correction routine. I have had occasional corrupt or lost DNG frames from non verified file copies. http://www.rawbot.me
 
You must log in or register to reply here.
Back
Top