DVX footage 24P (not 24PA), FCP 6.0.4 how to best remove pulldown & save HDD space?

rjcam

rjcam

Active member
DVX footage 24P (not 24PA), FCP 6.0.4 how to best remove pulldown & save HDD space?

Hi:
I shot 220GB of footage to tape at regular 24P setting on the DVX (doc film). I have the film edited to 2 hr 40 min. I calculated roughly need 36 GB of hard drive space for portions of clips I actually want to use and maybe 27GB at true 24P pulldown removed.

Have full licensed 6.0 suite upgraded to 6.0.2 and 6.0.4 versions. Have Cinema Tools and Compressor with it.
Time (overnights) not too important. Want best looking quality.
Should I use Cinema Tools within FCP to remove the pulldown? I did some tests and cannot believe how freaking bad that sh*tty pulldown made my footage look - particularly on interviewing people's teeth outdoors - tons of 'jaggies' and blocky errors. With C.T. pulldown removed tests it all looks great. And I have done 29.97 DVD exports on very very expensive analog and digi projectors and it still looks horrible and blocky in portions of the frame with aliased outlines etc... So the default 29.97 clips and timeline playback not just a digi / PC monitor error it's really there and a real problem.

Anyway, what I am asking really, is best way to do it - Compressor or C.T. - and more directly how I can save out new separate media files / clips of actual beautiful 24P footage that only takes the actual USED portions of the original clips in the timeline - not the entire original clip and full length b/c it means I need to hog up a lot more hard drive space and it would take so much more time to convert all that footage on my hard drive (15 hours of original tape source). AND since the cam was powered up and down etc... and some clips are 35 minutes long I imported into FCP, I bet from researching that the 2:3 cadence changes randomly, so little short portions of clips should convert much more reliably and not cross into new shots with a maybe different field order / cadence / voodoo / hiccup in it etc...

My ultimate plan is to master it to a true 24P DVD and then project it at true 24P with a smart projector and submit the doc with clean looking and professional 24P. I know how bad the 29.97 FCP auto setting sequence I used looks. What a mess.

Thanks!

RJ
 
Last edited:
I don't think you can do it correctly without having to re-edit the entire thing.

You cut it as 29.97; there's no way for you to know, and no way for Cinema Tools to know, on what frame your pulldown cadence begins in any given edited clip. And you need to know that in order to do a proper removal; if you don't, it'll be worse than what you already see. So even if you could export trimmed clips from the timeline, it would be a guessing game on each one.

And if you treat the entire 2hr 40mn program as a single clip, it's likely to be an even bigger mess, with some portions of the program coming out fine, while others won't, and you won't know until it's done exactly how that will play out.

Recutting it probably won't be as bad as you think, though, if you keep your original project for reference. An EDL export/import won't work because the frame numbering will be entirely different, but at least you can see which clips you used and generally where the cuts are.

(Because this is in the DVX100 section and not the FCP section, I should point out this would be a relatively easy fix in other NLEs, so if you're using something else and have the same problem, don't panic solely because of this post.)
 
Thanks for the response. I won't re-edit it b/c it's finished essentially, extremely complex and will take way too long.*
I've done some interesting HD 720 24PN edit conversions to SD 29.97 in the past... So I should be able to figure this out. Somehow.
What do you think of this idea I found online below?





If you're footage is already edited, you might try a solution that I've had success with. Export the sequence as a Quicktime Reference then run it through compressor with the ProRes codec, except in the settings turn frame controls on, and in the de-interlace pulldown select Reverse Telecine.*
Compressor automatically detects broken cadence so the process works even with already edited footage. This works for me, as I am often handed footage from the DVX100 shot in 24p (non-advanced) that has already been digitized. I edit the footage in a 29.97 timeline then follow the process that I described above. Cuts down dramatically on waiting time since you are only reverse telecine'ing your edited sequence (not all footage).


Only snag is I'm not sure how well it would work with a sequence with heavy effects or motion compositing, Probably not at all. Otherwise, hope this helps,
 
There's no reason not to give it a shot. If Compressor can detect a broken cadence, then maybe it'll work. Would be interested in knowing the result.

Going from 23.976 (720p 24pN) to SD 60i is easy, no muss, no fuss, not much to go wrong. Reversing the process properly is the trickier part.
 
Yeah going DVCPRO HD 24PN to 29.97 ntsc SD was not too bad. I had what i called 'black gaps' inserted in the sequence a lot where I'd have to manually slide video frames here and there where it forced gaps. not too bad.

Well, I just did this trick last night. Took awhile at best Quali 422 ProRes... *not good.*
It did make it 24P, and portions were right but lots of parts were all messed up where the stupid interlaced was now mashed in permanently. I am pretty certain it made these parts worse, not better.*
AND WORSE - it did not do what i really wanted. It merely made my whole part two sequence of 1 hr 9 min a 25GB single file. Not at all what I want. I want just the portions that were used in the sequence converted and saved as new separate files that can be re-connected to a new 23.976 / 23.98 whatever timeline that I can continue to tweak and edit and make a clean glorious 24P DVD with DVDSP later... *Ugh!
Is there any other way? I'd rather not have to convert each file at it's full length - yuck that's 220 GB of raw 29.97 DV tapes captured. Not cool.
I've been shooting for 17 years. It's nice to know I needed a degree in math to do a stupid and shoot and edit with FCP.

-- oh and one last comment... I don't like 'editors' (or even producers in fact) who wave their hands in the air and say they are 'not technical'. It's their job. Yup and I am editing my own content here, and edit pretty well. But this is kind of a stupid joke, that I cannot remove a 2:3 pulldown easily or correctly. I understand how it's all captured and recorded in camera. Well. But this whole business, I mean, I am sure there are lots and lots of people who'd want to do this and would greatly enhance their project (since interlace is so sh*tty). Why is this glossed over and not a real option with FCP? I think that's inexcusable. Yeah I shouldda removed the dumb pulldown after digitizing it (I guess I had to read all about this for hours and hours before I digitized footage 4 years back - really??). But really, is Cinetools really going to get all the potential cadence changes right? And furthermore, it's sad that only I shot the camera, for a week with 24P regular, 16x9 squeeze, TC regen / preset and had each tape increase it's timecode up one hour manually giving each tape a unique and proper TC). And there is CADENCE problem b/c what, I, turned a camera on and off? Are you kidding me? And Final Cut Pro / CT can't figure out the field order correctly and makes a huge mess of it all? The "A" frame? Locating them all manually to hand hold Cine Tools so it does freak out? really for 300 clips? joking.
 
Last edited:
It must not have detected the broken sequences, then.

Unfortunately, I don't think there's any other way. You need to do a proper pulldown removal, and it's looking like it needs to be clip-by clip.
 
Well, I did finally start this back up.
Have 311 files to convert.
Converted all but 60 successfully.
Original files (as noted above) shot at 24P (regular / 3:2 pulldown) SD 'squeeze' mode.
Used Cine Tools reverse telecine. Used batch reverse telecine which worked on all but about 90 files. I then went into CT and manually set the frame to the "A" frame. Was pretty easy to spot the A frame. And then it worked fine. The ones it skipped usually had a video drop out or hit in the first 1-3 frames but manually CT output these fine then.
HOWEVER, I noticed with 16 DV tapes originally digitized, the 60 files it can't convert are all clustered together in clumps in a row sequentially. AND they do not have 3 clean frames, 2 frames interlaced in FCP or CT windows like the easy to convert ones, but rather every frame junky looking interlaced or partially interlaced. There are NO clean 3 frames together on these bad files. The cadence cannot be detected, and either the cam went screwy while taping (I doubt it) or FCP is a piece of junk digitizing the tapes and can't read the embedded cadence etc... correctly and made a mess of the footage. Tape 1 for example, was digitized into 2 approx 30 min clips. The first clip is junk. The second clip was digitized through FCP correctly. All this footage shot on a nearly new DVX100b camera and digitized with a Sony DSR11 miniDV tape deck. All digitized at the same session. No settings were changed in the capture settings during ingest.
Someone else may help me using a custom command line de-interlacer using ffmpeg engine and some other tricks. They already de-interlaced my entire 2.5 hour export of this doc and the results were flawless and stunning, the graphics clean and the 'bad' footage in there processed okay although I will double check the bad footage parts tonight. The issue is that I want to be able to handle this the right way in FCP edit and not have to take a complete export MOV file every time to a 3rd party that I want to make a change.
I re-connected all the good 250+ files in my FCP project that are now at 23.98 progressive, made a new 24P sequence and cut and pasted 1.5 hours of the first half into the new progressive sequence. I have a ton of 'black gaps' in there in the vid and aud that I corrected by hand (a frame of nothing shoved in there), but the new project looks great and waaaay better than that interlaced horror show. I f*ing hate interlaced video. I only shoot P unless I am working for a boneheaded client who overrides my choice.
Next thing: convert these 60 stinkers, or worse have to go back to source tapes and try to re-digitize them all again.
Incidentally, most of the bad files were shot at night in lower light wide open and maybe even gained up to 3 or 6 dB. Not all, some day shots, but large majority night. Most of file shot in daylight.
 
Last edited:
Tried to re-digitize / capture using FCP 6.0.2 FW and my HVX200 camera the problem portions of the DV tape to no avail. The bad parts all look the same - complete garbage. I made a new posting on this new digitize attempt.
 
Samples UPLOADED now!
http://rjfilms.com/doc_samples/
These are all nice small clips, quick to download / analyze.
The 'Bad" file is the bad one - notice the ridiculous 'sawtoothing' on the interviewees teeth etc. It's on each frameWhat a horrible joke. Thanks interlace for nothing.
Then there is a regular pulldown added and interlaced in camera clip. Shot at 24P regular which added the 3:2 cadence standard pulldown.
And there is a pulldown removed clip that is now 24 Progressive with pulldown removed using CineTools where I selected the 'A' frame manually.

Here's that other post I made - here below with more details. WHAT DO I DO? With problem clips like the 'bad' one. Anyone have any clue at all how / why this happened? (No camera settings were changed, indeed back to back clips shooting switch sometimes between 'good' 3:2 pulldown and this mess that I have on the 'bad' clips.*

Here's the situation:
Shot about 16 tapes with same camera - DVX100b - over 9 days.
Camera always set to 24P regular (not PA). I was sole shooter.
Originally digitized with Sony DSR11 pro DV deck via FW to Mac / FCP 6.0
Used regular DV/NTSC 29.97 capture setting.
I have successfully removed the pulldown later on much footage using CineTools. It did about half correct ("reversed" folder) and skipped ("skipped" folder) about half. I was able to manually select the 'A' frame in CineTools and have it remove the pulldown and interlace on about half of those remaining original skipped files. All this footage looks fantastic (bye bye stupid interlace).
But the other remaining original and CT 'skipped' files I cannot get CT to do a successful pulldown removal on, these originally captured raw DV file are a visual mess. I have tried to re-digitize the bad tape portions now using my HVX200 camera via FW. But the footage is the same, same bad parts of tapes. I even tried to capture it in FCP using the 'Advanced' capture preset just in case the camera somehow tweaked out and was shooting 24PA mode. But I never switched it myself. Nothing works. And it's not 29.97 / 60i natively imaged camera setting either. It's definitely imaged with the camera at 24P (and yeah I know recorded to tape 60i). (another note, most of doc shot in daytime, but 80% of problems occur with night footage and camera usually shooting wide open and gain of 6dB - coincidence I assume?).
SO BIG QUESTION:
Why are portions of tapes showing horrible interlace "sawtoothing'" artifacts in EACH frame? With the normal recorded portions, it follows the 3:2 cadence correctly, showing 3 clean frames and two interlaced frames... But these bad parts of the tapes have interlace junk in all frames... on occasion I seem to find 1 in 10 or 12 frames looking clean but that's usually without movements. And I have checked in FCP viewer window at 100% (the way to monitor problems) and in CineTools.*

If people are really curious about this I can post samples. I would post:
24P imaged recorded at 29.97 straight out of camera and captured to FCP without pulldown removed / modified raw capture file.
24P reverse pulldown from the 29.97 captured file using CineTools resulting correct / clean fixed file.
and a raw capture problem portion that has junk - interlace - *in every single frame.

And yes, I have tried to manually select the 'A' frame with every possible combination, and CT makes it probably look worse.
Is there any explanation for this, or even better any possible solution? Could the field order reverse on the same tape? Could FCP be reading field orders wrong? I find it very suspicious that the Sony DSR11 and HVX200 camera both result in same garbage fully interlaced looking capture on those bad portions of the DV tape. This is affecting about 20% of my footage and I am trying to finish the output of this documentary and this is holding this all up.

Thanks!!!
 
ANSWER TO YOUR PROBLEM, NOT MINE!

I realized over a thousand*people*and counting have viewed this, which either means a lot of people have this problem (I Doubt), OR people are looking for the best way to solve 24P / edit / post workflow.

So I feel like I should post the way to do it here now:
#1 and important!: What this WILL do for you and your footage: It will get rid of that annoying rotten 'sawtoothing' 'jaggies' 'electric buzzy lo res' looking frames that pop up constantly and unevenly in your footage. It also will look more truly like cinematic real 24P not the 60i look and feel that recording to DV tape forced on your footage (although this advantage is more subtle and less important to me frankly - I want all those resolution lowering 'jaggies' *gone forever! OH and you get Waaaay better graphics rendering and output in FCP, and faster renders, smaller file sizes etc etc...
- Digitize all 24P (regular) footage with the regular NTSC/DV digitize setting in Final Cut Pro with Firewire hookup from cam / deck playing DV tapes (you don't have to stress about settings here and no, don't try to force a 24P frame rate or anything like that - that is very bad, because right now your footage is in FACT 29.97 / 60i and needs to be transferred correctly at 29.97 regular NTSC).
- ideally, DON'T EDIT your footage now. You can edit and fix it all later, but it's a bunch more work and could get a little confusing - but it's not THAT bad. I've done it now after editing for a 2.5 hr film. I am just saying it makes more work removing 'black gaps' in your timeline etc...
- Open app CineTools (which comes with FCP install), and run the batch REVERSE TELECINE on your folder(s) of footage that you imported into FCP.
- I like to set instructions to make 'New file (smaller)', and keep original. It's a good idea since you may want to go back manually to re-try the CT process. And this selection tells Cinetools to keep the original file and not touch it and make a brand new file on it's own. It's smart to do this, and it really is fairly fast. Select *'23.98' for frame rate - its defaulted to this, you only want 24 if there is no sound with the clip (more or less).
- Cinetools in this case will make 2-3 folders: "Originals", "Reversed", "Skipped". Originals: the original files you digitized are now there untouched (but moved). Reversed: File CineTools thinks *it successfully removed the pulldown and made new 23.98 files progressive! (They will look beautiful!). Skipped folder contains files CineTools could not process and remove pulldown successfully. Double check the 'reversed' footage looks clean, if there are any issues you can re-do those files manually but I had no trouble with processed files.
- NOW, for those 'skipped' files. All is not lost, not at all. I had 300+ files to process of about 16 hours total. I was able to manually process all those files correctly (all the files that weren't screwed up in camera - a fluke I have no idea how it happened, read a separate of mine).
What you do is drag those files one at a time into Cinetools. Cinetools shows you what each frame really looks like and shows the interlace (you can also observe this in FCP in the 'viewer'' window set to 100% size). Using Cinetools, move forward with the arrow keys a frame at a time. What you are looking for is this pattern. You want the middle 'good' frame (1 of 3). It's easiest to find this in a section of your clip with motion - even a head moving back and forth would be fine. What in practice this means is: move through a frame at a time. You will see two frames in a row that have interlace artifacts aka 'sawtoothing' and look wrong. After those two frames, you will see 3 clean frames with no interlace visible. You want to move to the middle frame that is clean there (the 2nd / middle of 3 consecutive clean frames), THIS IS YOUR "A" frame. (You are now actually monitoring the 3:2 pattern 3:2:3:2 which is 3 'good frames' and 2 'interlaced' frames (in fact they are all interlaced but that's a separate worry for this purpose). You can select your 'A' frame by leaving your viewer window on ANY 'A' frame at all in your whole clip. This is telling Cinetools manually this IS the correct A frame pattern and it will back and forward process the whole clip correct now. Click 'reverse telecine' to process it. Ensure 23.98 is selected. And again I like (file generator instruction): *New(smaller).

- Import all the Reversed files into a new FCP project and make sure the edit timebase matches to 23.98 *
Your new project is a beautiful 24P project with your footage as you wanted it in the first place
(Hooray for new native 24PN cameras that record to Solid State memory P2!!!)
Good luck and enjoy!!
 
Last edited:
Well, better, just shoot 24pA. Or use an NLE which can remove standard pulldown as well as advanced (i.e., Vegas, Premiere, Edius, Avid).


It also will look more truly like cinematic real 24P not the 60i look and feel that recording to DV tape forced on your footage
Click to expand...

24p in a 60i stream doesn't look like 60i. It still looks like 24p.
 
Great point David.
I was initially confused about selecting 24PA, but if you want the 24P to edit with - yes, 24PA Waaay better option. You can skip the whole Cinetools pulldown removal separate step, have FCP do it automatically on ingest with the correct setting, and enjoy better final resolution.
*
I guess this was a decision I made in 2006 when I did not fully understand the reasons to pick 24P or 24PA. It was not explained well in dumbed down language, and made it seem if I thought I would want to go to video / TV (? maybe?) as final delivery 24P reg was better. Totally false, especially now with the advent of widespread LCD / Plasma / DLP flat tvs that can skip all the 60i garbage by often displaying progressive playback and in fact sometimes display pure 24P without any pulldown inserted anywhere in the stream, and with *24P BluRay and true 24P dvd authoring control.

2nd point. 24P regular does not look like 60i (it looks a lot better), but it also is not the same as real 24P playback if you leave the pulldown in. Those are 3 different display / playback settings and none look the same. What I mean is 24P in a 60i stream still has 30 fps playback and cadence and the annoying 'sawtoothing' playback that true 24P (pulldown removed) does not have.

When I figured this all out, I was quite annoyed I picked 24P regular over 24PA. (24P reg also suffers some resolution loss when the pulldown is removed that 24PA does not get affected by) I was annoyed that the difference between the *was not explained well and as times have changed a bit and progressed in a better direction which makes 24P regular almost always a poor choice.
 
Last edited:
Two quick clarifications:
Cinema Tools worked great and would have done the whole prject just fine had in camera assets not got screwed up. This more or less assumes that you digitize with the setting to make new clip on detected errors while digitizing. I suppose your digitze pref is not set to make new clip on fcp and if your tape or camera had a cadence break and change it would spread two or more different cadences in a single clip you would have to process the clip with two separate a frames manually in cinema tools or break the original clip into separate new clips where each cadence break occurs and manually find a frame for each new clip.
- Second: in my case I switched camera to f2 scene file flou (for fluorescent lighting scenes) which I found out in post to my horror automatically took my dvx100 camera out of 24p recording and made those scenes filmed all at 60i whch causd me fits
 
Last edited:
You must log in or register to reply here.
Back
Top