Save MXF or just Import into FCP?

[dvInsight]

I am trying to see if I really need to save my data twice. I am not talking about backup, that's a given, but at the acquisition stage.

I am Mac based and shooting 24pn and currently have FCP 5.04. Now once a card is full I can use one of the logging tools to ingest footage, or copy it to an appropriately named and organized folder system. That would get all the MXF data in its raw format. Then I would still have to import into FCP, thus doubling my storage requirements before backup is even considered.

OR,

Can I just import directly into FCP into an organized bin structure I keep there? That is what I have been doing since it appears to be the most efficient. But I have read that when Apple unwraps the MXF file that it tweaks it in such a way that you get some frame doubling in FCP 5.0.4. Is this accurate, and if I upgraded to FCP 5.1.1 would it then make this workflow viable. By viable, I mean that nothing is going to bite me going forward.

Lastly, if I do the FCP import and then want to give my footage to a guy with PC based AVID system, is there going to be a problem there? I have heard that Apple's DVCPro Codec is only playing nice on Macs at this time.

All of this is frustrating, I understood MXF is a wrapper, but it seems that every vendor unwraps it differently and then renders it differently too. There is a need for an unwrapping shootout!

So if anyone has had a similar issue, I would appreciate the feedback.

Thanks,

Rob

[dvInsight]

Barry, Jarred, Evin, anyone, anyone?

Should this be moved to technical?

Thanks,

Rob

[mark.burton]

But I have read that when Apple unwraps the MXF file that it tweaks it in such a way that you get some frame doubling in FCP 5.0.4. Is this accurate, and if I upgraded to FCP 5.1.1 would it then make this workflow viable. By viable, I mean that nothing is going to bite me going forward.

Hi Rob,

Since you are shooting 720pn24 you should definitely upgrade to FCP 5.1.1. There was a major bug with this format in earlier versions of FCP. Note, any imported files from earlier versions will still display the issue in FCP 5.1.1, it was an import issue so the media files still have the problem. Files imported in FCP 5.1.1 will be fine:
http://docs.info.apple.com/article.html?artnum=303950

With regard to the workflow you have that seems fine to me as long as the orginal Contents folders are also backed up elsewhere after the import (which you said they are!)
--
Mark

[Hilarity]

WHAT is the "telltale" sign of these import issues? or rather what are the issuses?

[mark.burton]

Check out this thread (http://discussions.apple.com/message.jspa?messageID=2640037#2640037) for some first hand experience of the issue.

[BenB]

For working with MXF/DVCPRO-HD footage, you REALLY want to upgrade to 5.1.1 for more reasons than I care to list here.

[dvInsight]

Thanks all,

I understand. The issues and errors in FCP prior to 5.1.1 were real and repeatable, so I obviously will upgrade.

But after that, is there any negative to just importing the into FCP on a Level 1 drive array so that the data is duplicated twice. So I am covered in the short term until what is in the capture folder can be backed up.

In essence, I am trying to skip copying the MXF Data and folder over, and then importing it into FCP. My only hesitation is that I may be losing a future option in some way.

Realistically the MXF file is not like a RAW file in a digital camera, but I do understand that different NLE's import this data differently when it gets unwrapped. I just need to know if someone has gotten burned by this other than the issues in older versions of FCP.

Hope this makes sense.
Thanks,

Rob

[Jan_Crittenden]

Throwing away the MXF is analogous to throwing away your tapes. These are your master files. I would archive these first and foremost with your secondary copy being the Quicktimes made by FCP.

If something goes wrong you will wish you had the MXF. I have talked to enough people that threw away their masters that my heart sinks every time I hear someone contemplating the demise of the MXF. Three words: Don't do it!

Best,

Jan

[mark.burton]

You should absolutely keep the original MXF folders, think of them as your master tapes. In this format they are the most compatible with different platforms and NLE's.

[mark.burton]

Jan it seems we're following each other around this morning :-)

[Jan_Crittenden]

What time do you have there? ;-) It is 6:45AM here.

Thanks for backing me up here.

Best,

Jan

[mark.burton]

11.52am, I'm not quite as early bird as you!

[Jarred Land]

yes.. sometimes you can get a little glitch in the quicktime wrap when your conforming from the MXF, so always keep your masters.

[dvInsight]

Grrrrrr.

I was fearing that answer. Not from the point that I have tossed anything yet, but more to the ill affect of having that much data redundancy.

I come from a data background, so the purest in me would then say that the MXF has to be duped twice for redundancy, and then the same goes for the FCP workflow files.

Of course this is staying pure. Different projects will ultimately affect archiving depth.

Will FCP ever deal with MXF directly? Jan?

Thanks,

Rob

[Jan_Crittenden]

Will FCP ever deal with MXF directly? Jan?



I don't know, I don't work for Apple, they are the only ones that can really answer that.

Best,

Jan

[BenB]

I don't see a problem with the current QT conversion for P2 data. Firt up, it's less CPU overhead. dealing directly with MXF data is just going to cause more CPU overhead.

I've not heard anything, as a trainer, not that we ever know anything, but we do get "leeks", but nothing about Apple changing the current scheme of using the DVCPRO-HD QT codec.

[dvInsight]

Thanks all,

I will keep adjusting my workflow based on the project at hand. On jobs that others will edit or be involved in I will keep the MXF data, on my own projects that I will control 100% I may keep with my FCP and DVCPro Codec workflow.

Thanks,

Rob

[rgbuser]

Hi, dvinsight: Have you looked at the actual size of the files you are complaining about having to archive? A typical .mxf file is smaller than when it's wrapped in .mov, and the other - sound, icon etc. files are a very small fraction of a typical Contents folder. Why risk the heart-ache for a few meg per Contents folder?
cheers
Tom

[Dan Montgomery]

Jan's right, absolutely keep the original MXF! I know of one customer that ran into a hard disk formatting issue that limited the QT output file sizes! They thought they had everything in QT format and tossed the MXF to save space. Use your imagination what transpired in post.

[dvInsight]

After shooting both with P2 and tethered, I only see the advantage to saving the MXF for cross platform work flow. If I trust the DVCPro codec when going directly into FCP, which I do, I see no reason not to trust it going into FCP from the P2 card.

Again, this comes down to a data management decision. Back up the MXF 2X and my FCP Capture 2X equals 2X more backup than I need. When storage is 10X faster and cheaper, then maybe so. But by then we will all be shooting 4K Raw files at 120 fps and needing a 100X increasing in storage and speed. Argghhh.

I vote for a chisel and stone tablets.

Thanks,

Rob

[Brian Petersen]

I don't think you'd need to do the Quicktime 2X. It seems that if something happened to a quicktime file, you just go back to you MXF files and reimport. No need to backup the Quicktime files, right?

So if I'm understanding this correct, a proper and safe workflow would be have 3 sets of footage. Your original MXF files, a backup of the MXF files, and a Coverted to QT files to edit with. Correct?

[BenB]

Just something to think about...

Here's what I do. I copy the MXF files to my Media Drive. I import that into FCP, that makes the QT files in my Capture Scratch folder for FCP. I keep both. In case something happens, I have the original MXF data, which is you "original tape". That MXF file IS your originally shot on tape, don't trash it!

Then, cause I've got consulting clients who have, and I have, had drive crashes and were not prepared, no "reliable" back up. So, I have a 1TB dirve that does nothing but get turned on once a week, and all three of my drives; system, media 1, media 1, all get literal copies to that drive. Next time I crash (cause it's when, never if) I lose a few days at most. And with a click and drag, in 10 minutes, I'm up and running like nothing happened. I pay my house note off this work, I have a lot of my client's money tied up in this work, I can't afford to lose anything. So that 1TB drive is MORE than worth what I paid for it.

If you've ever lost the better part of a project you're being paid for, or is very dear to you, you know how sick it can make you feel.

Thus, I have three copies of all my footage at any one time. One option is to have 2 media drives, copy your original tape, er, I mean the MXF data to one, and have FCP use the other as it's scratch drive. So "when" a drive goes out, you don't loose your footage.

Just something to think about...

[dvInsight]

Brian, Ben,

There are several issues all going on at once here. One involves the differences between what is in the MXF file vs what is in a Quicktime version of the same thing. IF they are identical part two concerning storage goes away, since I have good workflow and data management strategies in place already. Lastly the differences between the MXF file and Quicktime file would obviously go away if Apple simply supported MXF files natively, which as Jan pointed out, Apple doesn't tell her anything, so I am not holding my breath for Apple to do that.

My post really started out by trying to figure out if I lost anything (I am talking color space, resolution, file intraoperability etc) other than meta data if I simply imported my clips directly into FCP and never bothered to save the MXF files. That never seemed to be answered here clearly, other than these (the mxf files) are like your negatives, raw files, or even your kids. All of those superlatives made the MXF files sound very important and valuable, but I still don't really know if there is any more, or maybe different is a better word, color and resolution information than what I get using Apple's FCP DVCPro Codec. From what I always understood, Apple is simply unwrapping the information exactly from the MXF file. Since the HVX is compressing to the DVCPro codec, it is what it is already. Right?

So back to data management. You actually have different issues here too. There are three components for the truly paranoid. I am among them. You have backup, redundancy, and archiving. When customers pay me to do something I worry until it is done. Done is when it has met the expectations and desires of the customers and played out its lifespan as defined by them. Then it is archived.

So, back to how many copies of what to keep. If the MXF files are gold then you really have no other choice but to have two copies, preferably archived on two different formats (DVD, DLT, Hard Drive, etc) stored in two separate and safe locations. Why, well bad things happen and usually in threes. Crap maybe you need a third (ok, I am stretching here).

So that is your first data set that you hope not to go back to. Next is a working layer of redundancy such as a level one raid, providing a mirrored copy of everything your actively importing and working on. So the level one raid is providing a safety net if your active drive dies. If that happens you can continue chugging along on the single drive that is left, but you are without a safety net until you can sucessfully replace and reimage the mirrored drive.

So great, you are safe right? You have a 2 backups of your MXF and you think you have it all together with your level one mirrored drive for your work setup. Not really, the thing you really need to do now is offload the mirrored stuff to another backup system at the end of the day. This backup may itself have levels of redundecy that I won't get into here, but you get the picture.

Why do you need this on top of everything else? well backups like this let you go back to any point in time and pick up the pieces from that point instead of going all the way back to the MXF files. If you are 20 days into an edit and some one doesn't know what a capture folder is and dumps it in the trash off of your level one mirror, they have just dumped off both drives simultaneously. Not to mention drive theft, power surges, and random acts of god. So that last backup really lets you go back to the day before the disaster.

At this point, you have at least 5 levels of data to deal with. So back to my first point, if the MXF and Quicktime are really the same, then I don't have to deal with the first two backups. All in all this is a tremendous saving in both time and money.

Thanks,
and yes I am that paranoid.

--

Rob

[Jan_Crittenden]

Hi,

My feeling is that the MXF is the interoperability file. You cannot take the QT file from Apple and expect it to work on the Avid. And for archive of footage in the long run, nothing will top the ability to search the MXF for Data. Keep in mind we are at the beginning of this transition and all IT productions and archive solutions have not been fully fleshed out, and the only recommendation that I can give is that the MXF files are the only ones that should be left standing for the archive.

Do you archive the finished production, of course along with it's components, but the one that will prove to be cross platform compatible will be the MXF.

My .02,

Jan

[VaricamLife]

If you are that worried about having a safe 'backup' or archive, then just take the footage over to DVC Pro HD cassettes. We're using the HVX as a secondary camera, or maybe its a tertiary one, anyway, so the 'tapeless' workflow doesn't really jive well for us. So I'm just making a copy of the MXFs because that is the 'RAW' format the HVX produces. However, I'm actually not using them at all. The rest of our project is a tape workflow doing offline and online edits. So I'm taking the HVX footage over to a tape and then loading it in from there so it will work out the exact same way as everything else in post workflow.

However, you don't need to load the materials like I mentioned. You could just go back out to tapes if you really are that worried about losing the information. Though frankly tapes degrade, even in cold storage. So if you put your MXFs onto DVD-Rs, hard drives, or whatever, and they too all degrade, then you're really not in much of a different boat. If you make even 2 backups of the MXFs then you're already ahead of the old tape workflows as how many people sit down and make direct duplicates of all of their tapes? But if you have 2 versions of the the MXFs in two different locations, you're actually using a tapeless workflow that is more secure than a tape workflow.

As for MOV vs MXF, just think about Jan's interoperability comments. These are backups you're talking about. So say you want to access the footage a few years later. Perhaps there is stock materials in there, perhaps you're re-cutting it to suit a new producer/distributor that wants to buy it, whatever, what if at that point you're no long on FCP? What if you've switched to Avid or another brand? What if someone wanting to buy the stock materials doesn't use FCP or Apple Macintosh? Why bind backups/archives to a proprietary wrapper, why not keep the universial one?

cheers.

[dvInsight]

Jan, Varicam life,

I hope you didn't see my post as negative. In fact I am quite the realist, and all of my issues would be resolved by either Apple supporting MXF natively, as I mentioned, or switching platforms (which would mean that I had arrived at room temperature). Since Macs can now do both I may temper my room temperature statement.

The workflow I outlined above is for my commercial clients who may not know if they are editing in FCP or Avid. On projects that I have control of from A to Z I think I may just go with a pure Quicktime based workflow.

My only big "unless" goes back to, how does the Panasonic DVCpro encoded video differ, if at all, from Apples rendition? I know that there is another thread digging around for this same info too. So hopefully Jan, or a Panny engineer could provide the definitive answer here.

And Varicam, I am thrilled to be moving away from tape. So, even though I consider it sometimes, my data background and dealing with the linear nature of tape makes it even more difficult for me to put back into my workflow. Right now a 63 minute D5 goes for about $150, which is the same as a couple of 80GB drives, or a single 200GB drive. So it is a wash as far as price goes, but I get the benefit of instant access when using the drives.

As Jan mentioned this is all a new workflow and solutions to data management will improve. Then again, I have a stack of AIT & DLT tapes, 100's of Zips & Jaz carts, early SCSI drives and arrays, and numerous Maxtor, Seagate, Western Digital raw drives, not to mention various firewire 400/800 products, and now sata. Hey at least I can say I don't have any 5.25" floppies, uhhh wait, I do have a few twiggy discs from an early Apple Lisa computer that still works.

I'm waiting for my implant, a few hundred petabytes. It ought to do. Right? :-)

All the best,

Rob