Where did some of my clips go? Very troubling.

M

Marshallarts

Well-known member
I just got back from a week long travel shoot. Each night I backup up my cards to disk image using FCP log and transfer. I backed them up independently onto separate drives.

After arriving back home I let log and transfer work over night to transcode and ingest to ProResLT. The next morning I noticed two hadn't transfered. Those two aren't even appearing in log and transfer and unfortunately one is a very critical long clip of an interview! I tried using ClipWrap but that program isn't recognizing it at all either!

I am able to open up the .MTS file using VLC. How can I rescue these files and more importantly why do we trust tapeless when something like this can happen!!

This is inexcusable for a supposed professional camera. I cannot go into the field, record footage where everything appears fine, only to learn one is corrupted. WTF?!

I tried Panasonic's useless AVCCAMRestorer software.. Good one Panasonic, great job making real crap programs that don't work. It says a file is missing from my disk image. I'm a loyal Panasonic customer, I love my HDX900. This really gets me mad.
 
Last edited:
If you have access to Premiere Pro, you might be able to import the MTS file. If you can, drop it in the timeline and export it using the DVD-Blue Ray settings, high quality render. You'll get and mp4 file that is every bit as good as the original.
 
Before you go blaming the camera, why not try putting the card back in and seeing if it works in the camera?

I find Log & Transfer to be less than 100% reliable, which is one reason I hate it. I think it's far more likely that L&T is failing, or the oft-documented problems that OSX has copying FAT32 volumes to the computer, than that you have a problem with your actual footage.
 
Stevedocmaker, thanks for the reply, unfortunately I do not have Premiere Pro.. Do you or Barry know any other programs I could try to do this?

Barry_Green said:
Before you go blaming the camera, why not try putting the card back in and seeing if it works in the camera?

I find Log & Transfer to be less than 100% reliable, which is one reason I hate it. I think it's far more likely that L&T is failing, or the oft-documented problems that OSX has copying FAT32 volumes to the computer, than that you have a problem with your actual footage.
Click to expand...

Well, the card has been formatted and reused many times since. On the road I carry 4x 32GB cards and backup to disc images nightly to be reused the next day. Not that your saying this but a camera manufacturer shouldn't expect someone to card 20 cards with them when on the road in case something like this may happen, seems to be awfully good cop-out excuse in their past since they're the ones pushing tapeless. If that were the case I'd stuck to tapes. BTW, I love your book and really respect your opinion. It's just very disappointing when technological "advancements" create problems.

Which transfer program do you recommend and use?
 
Marshallarts said:
Stevedocmaker, thanks for the reply, unfortunately I do not have Premiere Pro.. Do you or Barry know any other programs I could try to do this?



Well, the card has been formatted and reused many times since. On the road I carry 4x 32GB cards and backup to disc images nightly to be reused the next day. Not that your saying this but a camera manufacturer shouldn't expect someone to card 20 cards with them when on the road in case something like this may happen, seems to be awfully good cop-out excuse in their past since they're the ones pushing tapeless. If that were the case I'd stuck to tapes. BTW, I love your book and really respect your opinion. It's just very disappointing when technological "advancements" create problems.

Which transfer program do you recommend and use?
Click to expand...

You can copy the footage back to the cards if you made a complete image of the card and have the entire structure intact like you said you have, and there is not a problem with the disk image (which would not be the camera's fault)... Panny does not expect you to carry 20x cards on shoots (though the prudent thing to do is not format them before L&T and verifying the clips), but seriously, do you know how much tape you would have to carry to equal the recording time 128gb of AVCCAM footage... and you should not be reusing tape. These are not problems that come with the advancement of tape to digital workflows. Tapeless workflows are more stable than tape-based ones nowadays... and I have never had to send out a SD card reader to get its heads cleaned, never had an SD card get damaged by a overused reader at a post house, never had to send out a card reader for repair because the card I was transferring got jammed in the deck. Too many moving parts and problems with tape... There is a reason it is not just panasonic who has dropped tape. Why do you think everyone is going SSD nowadays?
 
Last edited:
Once upon a time I thought that going tapeless would reduce the number of problems, but ... sigh, it has not.

Nobody expects you to carry around the cards forever. But what I'm saying is -- the card is almost certainly fine. It's the transfer to the computer that is the issue, or the L&T process that is failing to recognize perfectly valid files. We've had complaints of bogus L&T performance here for the last six years, ever since it was first introduced. It isn't 100% reliable. Sometimes it just. doesn't. work. I don't know why. It's one of many reasons why I just won't use it.

For a verified transfer, the only program I know of that will do it is Imagine's ShotPut Pro. It has the ability to do a byte-for-byte verification of any copy. And when working with tapeless, that's what you need. You need to know that the copy has absolutely and completely succeeded. I've been trying to track down the source of the error for years, and the closest we've come is that Apple's OS just isn't as reliable with FAT32 as it is with other file systems, and doubly especially when using an external firewire drive. And since it seems like nobody is willing to truly identify where the problem is, my catchall solution is to say that if you use a data-verifying copy, you should avoid getting screwed. The copy errors might happen, but at least the data-verify process will catch them and give you a good shot at fixing it.

Soon the whole L&T problem will disappear (or, at least, it could). Panasonic is releasing a free plug-in for Quicktime that will allow any modern Intel-based Mac to read .MTS files just like Quicktime movies and will eliminate the need for Log & Transfer. Some folks may choose to c0ntinue using that workflow if they want to, or if they prefer ProRes or whatever, but I'm really really really looking forward to having a simple solution that will forever solve people's L&T problems.
 
Marshallarts said:
I just got back from a week long travel shoot. Each night I backup up my cards to disk image using FCP log and transfer. I backed them up independently onto separate drives.
Click to expand...

Hi Marshall, I got a couple ideas for you, but first can you help me understand your workflow a bit better. How are you making the disk images? Did you Lock the cards before putting them in your card reader? You said you use FCP log and transfer to do that, but as far as I'm aware, that is not a function of FCP.
If you do have disk images of the original cards, then you can copy that back to a card and put it in the camera as barry suggested, see if the files play in the camera.

Although you don't have premiere, Adobe offers free 30 day fully functional trials, so you could download it just to try out these clips.

Going forward I'd recommend you ALWAYS LOCK THE CARDS as soon as they are pulled out of the camera. When you mount a volume in OSX, it will start writing an index file to the volume. At times if a card is completely full when the OS tries to do that, clips can get corrupted. This happens on any non tape based media from any camera.

I'd also recommend using shotput pro to offload the cards. It automates the process and verifies the backups.

best of luck with your footage
 
Thanks to everyone for the suggestions and much needed help. I think I've managed to come through okay on this one. Let me recap what I've learned and respond to your replies in effort to help me.

I discovered something was wrong when the number of .MTS files in my cards didn't match the number of clips that were both transcoded and found by L&T. I came to learn I was half right. I first realized something was wrong when after my overnight tranfer 2 of my clips had a "!" next to them and weren't transferring. I then realized one of the clips in fact did transfer but still had a "!" next to it (?) and remained in the que. The other one didn't transfer at all.

I wanted to double check my clip count and so I mounted all my disc images and counted the .MTS files. My final transcoded clips were still 2 less, I knew the one that didn't transfer but after hunting found there was a .MTS file of a very important interview that appeared to have not transferred!

I tried Shotput Demo and Premier (as recommended by you guys) and nothing could 'resurrect' this crucial .MTS file. I tried restoring my disc image to my SD card and I couldn't get DiskUtility to do it!!! It keeps saying there is not enough room!! (Panasonic's useless program won't help because it doesn't even recognize disc images.) I then remembered I didn't think I cut at all during the interview yet there were 2x .MTS files. After viewing the untransferable .MTS file hoping nothing was that crucial on it I checked the transcoded ProRes clip from the clip before, crossed my fingers, and sure enough that single transcoded clip contains both .MTS files. Curious it even appears at a single clip in L&T, but when it gets to the point where the second .MTS file would begin the playback bar will continue but if you try to jump to any point which would reside in that last .MTS file the playback bar will jump to the very end and not allow you to jump to a portion in that file.

Maybe I missed the memo or the section in Barry's book saying long files span over multiple .MTS files (FAT32?---probably, I'll have to remind myself to look that up), but it appears I did not loose that important interview.

The other clip, however, I can't get to recover. Luckily it's a broll shot and I'm not wasting another second with it. It does show up in L&T, but when viewing (both in L&T AND the .MTS file in VLC!) there is a serious drop out. The clip was only probably 20sec and I remember taking it. There was no issue, I was on sticks, nothing bumped the camera, there is no reason why it would have specifically happened to that clip. But there is a dropout a few seconds in lasting a few seconds and L&T and ClipWrap can't transfer it (I didn't try shotput).

Have I regained my trust for tapeless?---Most certainly not!!

@Oliver Rush. Thanks for your help. But I disagree a bit with your defense of tapeless. I certainly know how much tape I would have to carry to equal recording time of tapeless. That's what I did for years! I've had entire luggage bags packed with giant Beta stock. And I most certainly would not reuse tape, unlike SD cards which I "reuse" every day. Traveling with tapes I don't need to worry about being prudent to ingest before I worry about reusing them. Because I don't reuse them. A traveling shooter isn't always able to verify the transfer of footage from a card and relies on the tapeless backup workflow unlike tapes which are there until you loose them. :)


@Barry. Excellent explanation, and I in fact was backing up to external FW800 drives. From now on I will use the data verifiable ShotPut program you and tacotim recommend.

If FCP allows working with .MTS files in the future do you see yourself transcoding to ProRes at all for editing?

@tacotim. I did not lock the cards..... :( I completely forgot, and now I do remember hearing in the past about that issue. I thought that issue may have been fixed, maybe not, but in truth I didn't lock it. But yes, L&T does allow you to create disc images. If you right clip on the card it will give you the option.
 
Fat32 has a 4gb size limit so a new clip gets started once it reaches 4 gb...I also discovered this by accident the same way you did. Normally I have always used shotout pro but recently had a client who did all of the downloading on his own, I had suggested using shotput but he instead would just do log and transfer right from the card, which I strongly discouraged as he never saved the raw mts. Anyway he was double checking the number of files on the card against the prores files and found what you did...that's when I figured out the 4gb limit, it just never had been an issue in my workflow.
 
happy to hear that the important stuff was recovered. I think you'll be happy with shotput pro. I've been using it for several years with p2 and RED files, and now with af100. I have yet to have a lost clip.
 
These problems you never or seldom bump into when on win7. Apple makes good computers but the os is still lacking when it comes to data management.
 
You must log in or register to reply here.
Back
Top