Clever, FREE pulldown-removal & transcoding workflow via AviSynth

[ydgmdlu]

Inspired by the work of SLoNiCK, I've developed a primarily Windows-based workflow that's completely free (and legal) and offers at least as good, if not better, results than anything else attempted so far. Although the first-time set-up is a minor hassle, I think that processing individual files will be less work than the other methods that have been proposed and tested (other than, of course, CineForm Neo Scene). There is a slightly convoluted way of getting this to work with Mac OS X, which I will explain later. We're using Avid DNxHD as the intermediate codec. It's comparable to ProRes; in fact, Apple created ProRes for FCP to compete with Avid's codecs for its NLEs.

What you need:


AviSynth 2.5.8 (http://avisynth.org/mediawiki/Main_Page)
TIVTC Filter (http://web.missouri.edu/%7Ekes25c/)
QuickTime Import Filter (http://forum.doom9.org/showthread.php?t=104293)
Haali Media Splitter (http://haali.cs.msu.ru/mkv/)
ffdshow tryouts (http://ffdshow-tryout.sourceforge.net/)
Avid QuickTime Codecs LE 2.0 for Windows (http://tinyurl.com/d24odv)
MTS file to process
My AviSynth script


For your convenience, I have packaged all of the necessary files into a ZIP archive that you can download here: http://www.filedropper.com/avchdworkflow

Set-up instructions:


Install AviSynth.
Copy TIVTC.dll and QTSource.dll to C:\Program Files\AviSynth 2.5\plugins
Install Haali Media Splitter. During installation, make sure that "Enable MPEG-TS support" is checked.
Install ffdshow tryouts. During installation, make sure that ffdshow is set to decode H.264 video.
Install Avid QuickTime codecs.


To remove pulldown and transcode:


Open script.avs in Notepad.
Change what's between the quotation marks in the first line to match the name of the file that you want to process.
Change what's between the quotation marks in the last line to whatever you want to name the output file.
Open script.avs in Windows Media Player. WMP will tell you that it doesn't recognize the type of file, but tell it to try to play anyway. Check the box so that you won't get the same warning every time.
A dialog box will soon open asking you to specify the QuickTime encoding configuration. Just change the compression type to "Avid DNxHD Codec," depth to "Best Depth," and resolution to one of the 1080p/23.976 options (your choice). Click OK.


The dialog box will close, and WMP will slowly play the video. Do not interrupt the playback. Once it reaches the end, DO NOT close WMP. The QuickTime encoder will still be encoding the output file, and it NEEDS the input file to be open while it's working. Unfortunately, there's no progress bar of any sort. However, you can monitor the progress by opening the destination folder in a Windows Explorer window. Set the view mode of that window to "Details." If you keep pressing the F5 key, you'll notice that the size of the output file will be rapidly increasing. Eventually, that number will stop growing, at which point the process is complete. You can play the video in QuickTime to review the result.

If you install the Mac versions of the Avid codecs, you can then edit the transcoded footage in FCP.

Right now, the directions above are a proof-of-concept. With the process, I was able to successfully transcode SonicStates's cat clip (the example.MTS in the package) without any artifacts. There are ways to automate the process a little more; for example, SLoNiCK showed how you could write a batch script to generate AviSynth scripts for all of the MTS files in a given folder. I imagine that the entire process could be automated if a programmer could write a front-end that handles all of the steps. I don't think that it would be very hard, but I'm not a Windows programmer myself.

If you don't have easy access to a Windows computer, then there is a way to do what I described. There's a free application called VirtualBox (http://www.virtualbox.org/) that would allow you to create a virtual Windows XP PC inside of Mac OS X. It's a lot like Parallels or VMware Fusion, except that it's free. All that you would need is to "borrow" a copy of Windows XP to install, and you still wouldn't have to pay anything. The virtual machine would be set-up to share a folder with the host Mac computer, and you would use that shared folder as your transcoding destination. In other words, transcode in Windows and instantly switch back to Mac to bring the files into FCP. This would actually be a viable workaround for any workflow that demands Windows, such as the current version of Neo Scene.

Let me know if you have any questions, and please give me feedback on this.

[Isaac_Brody]

Hey thanks for putting this together. I'm not really using windows anymore but would love to hear how this works for people. Virtualbox sounds like an interesting workaround and I'll check it out.

[ydgmdlu]

I've also uploaded the transcoded version of example.MTS that was processed with my method. Download it here: http://www.filedropper.com/example

You'll need to have the Avid codecs installed in order to play it.

[Norbert]

I have set up a similar workflow for CS3 but without the pulldown bit, because I am in PAL land.

All I needed to install was Avisynth, PremiereAVSplugin, Haali Media Splitter and ffdshow. Then I use this batch script to generate avisynth scripts for all MTS-files in a given folder:

set pfs_path=.\
for /f "tokens=*" %%a in ('dir/b/s ^"%pfs_path%\*.mts^"') do echo directshowsource("%%a") > %%a.avs
exit

Then I just edit the avs-files. They play kind of slow so I installed the trial version of MPEG Pro. With that I can edit the original MTS-files with ease and then replace them with the avisynth scripts before final output.

I just wanted to confirm that this method works great, and it helped me in my decision to keep waiting for the GH1.

[Polanski]

This works great, though I can't use the quicktime in Edius... It shows nothing in Edius...

[ydgmdlu]

I recommend encoding with a different intermediate codec, in that case. But you could also try not using an intermediate codec at all and just importing the AVS script into Edius. But if you do that, you must delete the last line from the script, so that it doesn't attempt to output a QuickTime when you open it!

[Isaac_Brody]

I downloaded the example. I'm still seeing a chroma error when scrubbing through. Six frames in around the eyes. See it?

Though most people are not going to notice it at all.

[ydgmdlu]

Yes, I do see it when I'm going frame-by-frame. It's not too bad, but I agree that Cineform is better (which is to say, smoother). Personally, I'd buy Cineform and use it instead, as it's MUCH easier, faster, and better compressed. My workflow is designed for people who are too cheap for it. :-)

I'm going to post a video, soon, demonstrating how gloriously smooth the VirtualBox workaround is, in case nobody wants to wait for Cineform to fix their Mac version. Stay tuned.

[SLoNiCK]

ydgmdlu

Why did you choose TIVTC? Looks like it makes some kind of smart adaptive deinterlacing and then removes duplicated frames.

Polanski

This works great, though I can't use the quicktime in Edius... It shows nothing in Edius...

Remove this line from ydgmdlu's script:
QTOutput("example.mov")
as long as you are not interested in .mov, save script, drop into VirtualDub (it's free - http://www.virtualdub.org/ ), go to "Video - Compression" menu (or press ctrl+p), select codec (CanopusHQ), go to "File - Save as AVI" (or press F7) and save the .avi. The result will fit Edius quite well.

[ydgmdlu]

ydgmdlu

Why did you choose TIVTC? Looks like it makes some kind of smart adaptive deinterlacing and then removes duplicated frames.

Polanski

This works great, though I can't use the quicktime in Edius... It shows nothing in Edius...

Remove this line from ydgmdlu's script:
QTOutput("example.mov")
as long as you are not interested in .mov, save script, drop into VirtualDub (it's free - http://www.virtualdub.org/ ), go to "Video - Compression" menu (or press ctrl+p), select codec (CanopusHQ), go to "File - Save as AVI" (or press F7) and save the .avi. The result will fit Edius quite well.
TIVTC is not a smart adaptive deinterlacer. It's a comprehensive inverse telecine solution. It finds the duplicated fields generated by pulldown and properly reconstructs the original 24p (or 25p) signal. It only uses smart adaptive deinterlacing when it finds material that wasn't telecined (i.e. native NTSC video).

TIVTC is actually reputed to be the best inverse telecine filter for AviSynth. It'll work far better than your script, I think, since the filters that you use are blind to whatever pulldown pattern the GH1 actually employs. In other words, your script might actually weave and select the wrong frames. Just think about it.

[SLoNiCK]

ydgmdlu

The pulldown removal algorithm itself is very simple and periodic (applied to every other 5 frames). You can write it in basic AviSynth functions in less than dozen strings (so I'm a little bit confused about some 'special proper pulldown removal' that everyone is looking here). So the only thing that any 'automatic pulldown remover' has to do is to guess the period of pulldown. If it misses then you'll get still very interlaced video on output. It's hard not to notice it. But if it hits then you'll get quite nice progressive result without any sight of interlacing (at least on such clean video as that kitty). So I've got curios why there are still some slightly noticeable interlacing artifacts after TIVTC.

And heading back to GH1 video... I've tried my script with about 20 clips from different sources, and they all started with the same pulldown pattern. So it's safe to use with raw video from camera. More to say, if you cut clips on I-frames the pulldown pattern won't change. The only case when this script can fail is processing of video that was randomly cut. For example when someone edits GH1 footage in 60i project, outputs it to single file and wants to get 24p out of it.

[ydgmdlu]

If your script really works 100% of the time, then I would recommend it over mine, since it would seem to process the video a lot faster. But I'm guessing that even then, it's no match for CineForm or perhaps even VoltaicHD. But again, our AviSynth solutions are designed to be free...

[David Newman]

Update for you on CineForm pulldown removal. The Mac version of Neo Scene (and HD/4K) now support a fix for the GH1 pulldown, so work just like the PC version. We hope we've priced this tool low enough to make it the obvious choice or Mac or PC.

[DrBlaz]

ydgmdlu: the link http://www.filedropper.com/avchdworkflow does not seems to work

the avisynth code for GH1 pulldown removal is just:

Telecide(guide=1,post=2)
Decimate()

this outputs a 23.976 perfectly pulldown removed video to virtualdud . Or you can even load the script in premiere to edit the video directly avoiding the virtualdub saving-to-disck step.

But, if you preffer to save the video for editing, better do it with a lossless codec, try these free codecs:

http://lags.leetcode.net/codec.html
http://neuron2.net/www.math.berkeley.edu/benrg/huffyuv.html


not as good as cineform codec, cause the resulting files are bigger, but apart from that are excellent codecs!

[Schmoe]

Also, comparing the Neoscene result to the AVISynth result, the Cineform just looks a bit sharper. I'm not sure what kind of fairy dust they are sprinkling in over there at Cineform, but Neoscene offers a better image after pulldown than both AVISynth and TMPGENC Express(although TMPGENC is very close).

[John Caballero]

Do the 24p pulldown files from Neo Scene work with Edius 5?

[ydgmdlu]

ydgmdlu: the link http://www.filedropper.com/avchdworkflow does not seems to work
Thanks for letting me know. I guess FileDropper deleted the file after people stopped downloading it. I'll re-upload it soon.


the avisynth code for GH1 pulldown removal is just:

Telecide(guide=1,post=2)
Decimate()

this outputs a 23.976 perfectly pulldown removed video to virtualdud . Or you can even load the script in premiere to edit the video directly avoiding the virtualdub saving-to-disck step.
Basically, it works the exact same thing that my script does. But having choices is nice.


But, if you preffer to save the video for editing, better do it with a lossless codec, try these free codecs:

http://lags.leetcode.net/codec.html
http://neuron2.net/www.math.berkeley.edu/benrg/huffyuv.html

I highly doubt that people here want to work with losslessly compressed files. They are simply too large. That's why I didn't recommend a lossless codec in my package.

[David Newman]

Do the 24p pulldown files from Neo Scene work with Edius 5?

I have to admit I never tried Edius. Neo Scene uses standard codec APIs like Video for Windows and DirectShow, if Edius uses either of those it should work. With the free trial you will find out pretty quickly.

[SLoNiCK]

John Caballero

Do the 24p pulldown files from Neo Scene work with Edius 5?

It accepts this clip on timeline, but looses framebuffer on Q6600@3200 with 85% CPU load. I've tried different quality setting and different codecs for Edius project, but didn't succeed.

Schmoe

Also, comparing the Neoscene result to the AVISynth result, the Cineform just looks a bit sharper.

I've noticed only the difference in red channel level. Left - avs, right - neo.
http://www.dvxuser.com/V6/attachment.php?attachmentid=11223&stc=1&d=1243720029

ydgmdlu
DrBlaz

According to mans, both Telecide(guide=1,post=2) and TFM(order=-1, mode=5, PP=7) make some adaptive deinterlacing thus showing some artifacts.

I tried to create script to a) throw out interlaced frames after DoubleWeave, b) to remove dublicated frames. Here what I made:

directshowsource("_your_clip_.MTS")
AssumeTFF().doubleweave()

x01 = selectevery(10,0)
x02 = selectevery(10,1)
x03 = selectevery(10,2)
x04 = selectevery(10,3)
x05 = selectevery(10,4)
x06 = selectevery(10,5)
x07 = selectevery(10,6)
x08 = selectevery(10,7)
x09 = selectevery(10,8)
x10 = selectevery(10,9)

CmbTrsh = 2

y01 = ConditionalFilter (x01, x02, x01, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y02 = ConditionalFilter (x02, x03, x02, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y03 = ConditionalFilter (x03, x04, x03, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y04 = ConditionalFilter (x04, x05, x04, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y05 = ConditionalFilter (x05, x06, x05, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y06 = ConditionalFilter (x06, x07, x06, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y07 = ConditionalFilter (x07, x08, x07, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y08 = ConditionalFilter (x08, x09, x08, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y09 = ConditionalFilter (x09, x10, x09, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y10 = ConditionalFilter (x10, x09, x10, "IsCombedTIVTC(CmbTrsh)", "equals", "true")

Interleave(y01,y02,y03,y04,y05,y06,y07,y08,y09,y10 )
TDecimate(mode=0,cycle=10)
TDecimate(mode=0,cycle=9)
TDecimate(mode=0,cycle=8)
TDecimate(mode=0,cycle=7)
TDecimate(mode=0,cycle=6)
TDecimate(mode=0,cycle=5)

It uses TIVTC for guessing interlaced frames and dups removal. Note "CmbTrsh = 2". If you're getting some wrong result then try to play with it. It is interlaced frames detection threshold level, range 0-255. It can go mad on noisy video with such low level.

[David Newman]

The color difference are likely due to a YUV to RGB color conversion using 601 rather Rec.709 color math. Neo Scene's conversion remain in the source YUV color space, no RGB conversions are used, so Neo is likely the correct output. We don't avoid RGB just for color and quality reasons, we find it is a faster also.

[Schmoe]

Here's what I've tried so far to get a decent pulldown reversal out of AVISynth and the results of each:

*AssumeTFF().DoubleWeave().Pulldown(0,2) : Video size come out at 1920 x 1088, lots of interlacing/pulldown artifacts

*AssumeBFF().DoubleWeave().Pulldown(0,2) : Video size come out at 1920 x 1088, lots of interlacing/pulldown artifacts

*Telecide(guide=1,post=2).Decimate() : Video size come out at 1920 x 1088, but video has NO interlacing/pulldown artifacts. Video is softer than Neoscene, but this is possibly a result of the switch to 1088 height.

*Slonick's long script above: Video size come out at 1920 x 1088, but video has NO interlacing/pulldown artifacts. Video is softer than Neoscene, but this is possibly a result of the switch to 1088 height.

Not sure what is causing the video to be blown up to 1088. I have tried using both ffdshow-20020617, as well as the newest alpha version ffdshow-20041012 with the same results.

[ydgmdlu]

Schmoe's results confirm my suspicions that SLoNiCK's original script was matching and weaving the wrong fields together. It's because the algorithm is a "dumb" or "blind" one. That's why you need to use an intelligent fields matching algorithm, like TIVTC or DGAVCDec.

[DrBlaz]

why you get 1920x1088?? avisynth or telecide does not change the size, it seems you have a problem with directshow, probably it applies some filter or format conversion before serving to avisynth.

I'm having a directshow problem too, when using directshowsource, with latest ffdshow drivers installed, I'm getting the correct size, but also interlaced artifacts in some frames, but by using DGAVCDec the pulldown removal seems to be perfect so far:


avcsource("00009.dga")

Telecide(guide=1,post=2)
Decimate()

info()

[Schmoe]

Eureka! You hit it right on the head DrBlaz. I had a codec that was overriding FFDShow and decoding the AVCHD files. After combing through Graphedit and uninstalling/reinstalling some things I tried the tests again.

Using the following in AVISynth:
*AssumeTFF().DoubleWeave().Pulldown(0,2) : Worked perfectly, results identical to Neoscene

*Telecide(guide=1,post=2).Decimate() : Worked perfectly, results identical to Neoscene

*Slonick's long script above: Worked perfectly, results identical to Neoscene

So any of these methods will produce nice visual results, although I still need to test motion in the files to make sure there are no errors there.

Given all the different AVISynth options, I think I would go with Telecide(guide=1,post=2).Decimate() just for simplicity....

Now...does anyone have any good suggestions for an editable intermediate HD codec? That's another plus for Neoscene... it includes a great HD intermediate codec in the package.

[SLoNiCK]

Schmoe[b]

AssumeTFF().DoubleWeave()
Can you upload first 20 frames of this script outpu on your system?

...as the newest alpha version ffdshow-20041012...
Man, where are you digging them from? :)
http://www.free-codecs.com/download/FFDshow.htm

[b]ydgmdlu
DrBlaz

Well, we all have choice. You can spend some time playing with pulldown(n,m) parameters. When you have right ones - I bet it will work on all footage on _your_ system. At least until you change codecs and splitters.
You also can rely on some "smart" tools. For exapmle TFM and Telecide convert original "aa, bb, bc, cd, dd" sequence into "aa, bb, c'c', c'c', dd" and remove "c'c'" dups with Decimate. The problem is that c'c'!=cc cause c'c' is reconstructed via some complex deinterlacing algorithm. It adds more functionality to TFM and Telecide letting them work even with semiblended or improperly resized frames, but we don't need it.
My last "smart script" works a little different. It converts "aa, bb, bc, cd, dd" to "aa, ab, bb, bb, bc, cc, cd, dd, dd, da", then replace interlaced frames with progressive neighbours "aa, bb, bb, bb, cc, cc, dd, dd, dd, dd" and then removes 6 dups out of the sequence. But it stil relies on some "is-this-frame-interlaced" guessing that doesn't work 100% right. That's why I'm still for "blind way".

[Schmoe]

What format do you want the 20 frames in Slonick? I think the only HD format I currently have installed is the Cineform codec.

Also, which of these 3 methods do you believe would be most accurate?

[SLoNiCK]

Schmoe

Sorry, I started my post before you mentioned that problem had been solved, so 20 frames are not necessary.

I vote for "blind" way. I tried all 3 "smart" methods on video session filmed by SonicStates and have found issues for all of them. It's not like something really troublesome in original footage but they usually have tendency to jump out on color grading or VFX.
For the "blind IVTC" - it's hard to miss wrongly IVTCed clip :)

NeoScene is interesting solution. But its benefits lie mostly out of "low-end indie" workflow. Crossplatform? We are mainly stuck on mac or win, not both of them, we rarely have to transfer timeline edits insted raw footage (cut in Vegas, CG in Color? Brrr...). 10 bpc? Nice feature to acquire video from HD-SDI, but for VDSLRs and inexpensive camcorders we have only 8bpc. I don't want to say they useless, but everyone should look in mirror and ask himself: "Are they usefull for me?"

[David Newman]

CineForm should run file on a P4 on up, more likely an issue with the NLE you are using (we can't optimize for everything), how is desktop playback for you? There is a 10-bit advantage on 8-bit sources, it is subtle and relates to compression, color space conversions, and down stream grading in tools that can handle more precision (FCP, Premiere Pro, AE, Combustion etc.)

[SLoNiCK]

David Newman

how is desktop playback for you?
Nearly this way:
http://www.dvxuser.com/V6/attachment.php?attachmentid=11227&stc=1&d=1243790459

I have NeoScene 1.2.1 trial and Neo player 4.0.2.211 installed. Works stable but eats half of my CPU just for playback in best case (with VDub). Edius and WMP can't play video smoothly.

[DrBlaz]

however, cineform codec is 4:2:2 , and the free Huffyuv and Lagarith lossless codecs can compress 4:4:4

This is a big advantage if you want the best quality

[David Newman]

Odd, VirtualDub is the slowest here. Could point to a graphics card issue if you get a non-accerated surface. Try the settings control for Desktop Playback Fast (in Start->Programs->CineForm->NeoPlayer.)

[David Newman]

however, cineform codec is 4:2:2 , and the free Huffyuv and Lagarith lossless codecs can compress 4:4:4

This is an big advantage if you want the best quality

We also have 4:4:4 and 4:4:4:4 codec a 12-bit precision at significantly smaller sizes than 8-bit lossless codecs. So it all depends what you are looking for, such as price, performance, data rate, precision, compression quality, compatibility, metadata and archivabilty. We do very well on on many of those, we are just not free.

[DrBlaz]

sure, I think cineform codec is one of the bests codecs available, but we are talking about the version shipped with neoscene, its the 4:2:2 version.

For maximum lossless quality, I would chose huffyuf or lagarith because they are 4:4:4, the file sizes are bigger of course, but HDs are cheap.

And, for people who don't want big files and don't mind some compression, ffdshow offers MJPEG and other codecs also 4:2:2




We also have 4:4:4 and 4:4:4:4 codec a 12-bit precision at significantly smaller sizes than 8-bit lossless codecs. So it all depends what you are looking for, such as price, performance, data rate, precision, compression quality, compatibility, metadata and archivabilty. We do very well on on many of those, we are just not free.

[David Newman]

This is a GH1 forum, 4:4:4 would be overkill lossless or not, for a 4:2:0 source. Hard drives might be cheap but workflow and editing performance, transfer and backup times are are impacted. There is always more complex story. There is value in a Neo Scene purchase over the free solutions, it is just not for everyone.

[SLoNiCK]

David Newman

Try the settings control for Desktop Playback Fast (in Start->Programs->CineForm->NeoPlayer.)

Tried to. Found that I don't have them. So I uninstalled both NeoScene and Neo player, cleaned registry, installed back only NeoScene and here it is! 30% CPU load in both VD and WMP with full quality. But still can't play clips in Edius...

[nugai]

ydgmdlu: the link http://www.filedropper.com/avchdworkflow does not seems to work

Thanks for letting me know. I guess FileDropper deleted the file after people stopped downloading it. I'll re-upload it soon.

I stumbled across this forum looking for a 3:2 pulldown removal tool. Unfortunately, the FileDropper link seems to be broken and doesn't work anymore.

The software required is described above and can be downloaded individually, but the script seems to be the gem that holds it all together. Could someone please post the contents of the "script.avs" file in this thread (preferably wrapped in "code" tags)?

Thank you in advance,

-nugai

[dippytwitty]

4:4:4 would be overkill lossless or not, for a 4:2:0 source.

I hate to stick my nose in here, but the logic of that statement escapes me - if we're talking about an INTERMEDIATE codec, then the intent is to allow generations of edits, which means the original source is irrelevant, since you could be doing all kinds of coloring, fx etc. at each generation, that would benefit from the retention of fidelity. Isn't that the idea your product line is based on??

That said, I'm not suggesting a lossless codec over CineForm, I just think the reason I like & use CineForm is that I can rest easy knowing I can do multiple generations with (relative) safety. And as you mentioned, if I needed 4:4:4 I could just go ahead and buy it too.

Dip

[David Newman]

I hate to stick my nose in here, but the logic of that statement escapes me - if we're talking about an INTERMEDIATE codec, then the intent is to allow generations of edits, which means the original source is irrelevant, since you could be doing all kinds of coloring, fx etc. at each generation, that would benefit from the retention of fidelity. Isn't that the idea your product line is based on??

That said, I'm not suggesting a lossless codec over CineForm, I just think the reason I like & use CineForm is that I can rest easy knowing I can do multiple generations with (relative) safety. And as you mentioned, if I needed 4:4:4 I could just go ahead and buy it too.

Dip

Of couse, if you know you can benefit from 4:4:4 for FX elements then go for it, there nothing better. Most customers are concerned about preserving source quality, so I typical recommend 4:4:4 for film, Red, Genesis sources, etc.

[mutabot]

Hi all,

Got my GH1 recently and been playing around with different ways to remove the pulldown from 180p24 footage. Strangely none of the proposed Avisynth + VirtualDub solutions do work for me.

I even went ahead and installed everything fresh including OS (Win7 in my case) -- still same result.
I'm using the latest versions of AviSynth (258), ffdshow (beta7_rev3154_20091209), MatroskaSplitter, VirtualDub.

I'm loading .avs files and inspecting them in VirtualDub. Same result if I'm re-rendering video to .avi using VirtualDub...

One thing that does seem work is to re-render the video to .avi using Vegas and then use VirtualDub's built-in IVTC filter.

--
Michael

[mutabot]

Ok, apparently I've had some issues with Win7 and ffdshow.

Now, as for the AviSynth script. Looks like it is a best keep secret, or some sort of conspiracy going on around extracting 24p out of 60i... Anyways, I've spent some time figuring it out, and here is what I've got:

----
directshowsource("xxxxx.MTS").AssumeTFF().SeparateFields()

x0 = selectevery(10, 0,1).Weave()
x1 = selectevery(10, 2,3).Weave()
x2 = selectevery(10, 5,6).Weave()
x3 = selectevery(10, 8,9).Weave()

Interleave(x0,x1,x2,x3)
----
That's all! Simple!

Well apart from that the x2 can be defined as

x2 = selectevery(10, 5,6).AssumeBFF().Weave()

But, looks like it works either way. At least I do not see any interpolation artifacts in my test footage. As opposed to other methods where I see artifacts.

Not sure if any Decimate() is required, maybe somebody can contribute on this regards.

[Blackout]

this pulldown removal issue on the PC is driving me insane as to which is the best and what the losses are.. can someone give a complete list of various ways from free to expensive to do it correctly?

[swyzlstyx]

this pulldown removal issue on the PC is driving me insane as to which is the best and what the losses are.. can someone give a complete list of various ways from free to expensive to do it correctly?


Doesn't get much easier than NeoScene: http://www.cineform.com/neoscene/

It's fast, it's stupid easy, and the fact it removes the pulldown is just a side benefit. The best benefit is it transcodes to 4:2:2 Cineform AVI...which you can edit on a Commodore 64. Even Sony Vegas chokes on AVCHD on my quad-core PC...AVC simply sucks for editing.

Cons: It cost money...but so does the GH1.
Pros: It makes me money...but so does the GH1.

[mutabot]

this pulldown removal issue on the PC is driving me insane as to which is the best and what the losses are.. can someone give a complete list of various ways from free to expensive to do it correctly?

Actually, algorithm-wise, there is only one way... And there is no magic behind it. As to pixel format, with free approach you can choose your output pixel format and compression as you like.

Look, I know that once we are in the land of post-processing you have to pay money for software, and it seems the more you pay the better software becomes.
In the case of pulldown removal, there is only one algorithm. Point. Does not matter how much money you pay, the output quality re pulldown will be the same for proper removal.

This is the algorithm:

x0 = selectevery(10, 0,1).Weave()
x1 = selectevery(10, 2,3).Weave()
x2 = selectevery(10, 5,6).Weave()
x3 = selectevery(10, 8,9).Weave()

It makes 4 frames out of 10 fields, picking the right fields to build each progressive frame.

To use that, one needs to install: (http://avisynth.org/)
1. Avisynth
2. ffdshow tryouts
3. Haali Media Splitter
4. VirtualDub

NOTE: Win 7 users must disable built-in mp4 decoder for this to work.

Then cut the below code and paste it into .bat file i.e. 24p.bat
Launch this file in the folder with your .mts files, it will process all files that are 1920x1080 and make 1080p .avi file for each corresponding .mts file.

Note that in the .bat below compression is commented out and output format set to 11 that corresponds to 4:2:2 YCbCr (YUY2)

You can set these values to your liking. To find out what value corresponds to what, open virtualdub, set Video->Color Depth (or/and Compression), hit Ctrl-S, open saved file, locate SetOutputFormat and/or SetCompression, paste corresponding values to your batch file.

The file below does not set compression and sets pixel format to to 4:2:2 YCbCr (YUY2)

---- 24p.bat begin ----

for %%f IN (*.mts) DO (
echo directshowsource^("%%f"^).AssumeTFF^(^).SeparateFields^(^) > %%~nf.avs
echo Assert^(width^(^) == 1920, "Not a 1080p24 source"^) >> %%~nf.avs
echo x0 = selectevery^(10, 0,1^).Weave^(^) >> %%~nf.avs
echo x1 = selectevery^(10, 2,3^).Weave^(^) >> %%~nf.avs
echo x2 = selectevery^(10, 5,6^).Weave^(^) >> %%~nf.avs
echo x3 = selectevery^(10, 8,9^).Weave^(^) >> %%~nf.avs

echo Interleave^(x0,x1,x2,x3^) >> %%~nf.avs
)

for %%f IN (*.avs) DO (
echo VirtualDub.Open^("%%f"^); > %%~nf.script
REM echo VirtualDub.video.SetCompression^(0x636c6c63,0,1000 0,0^); >> %%~nf.script
echo VirtualDub.video.SetOutputFormat^(11^); >> %%~nf.script
echo VirtualDub.SaveAVI^("%%~nf.avi"^); >> %%~nf.script
echo VirtualDub.Close^(^); >> %%~nf.script
)

for %%f IN (*.script) DO (
"C:\Program Files\VirtualDub\vdub.exe" /min /x /s %%~ff
)

for %%f IN (*.script) DO (
del /Q %%f
)

for %%f IN (*.avs) DO (
del /Q %%f
)

---- 24p.bat end ---

[tonpole]

Not to bring up a dead thread but none of these methods work. Is there still no solution for PC other than neoscene and its bloated output?

[Svart]

When you deinterlace and then reconstruct 24 real frames from a couple I frames and a bunch of P frames you don't expect your file size to get bigger?

[tonpole]

Not 500x bigger, no. Doesnt matter anyways, voltaicHD crashes, neoscene is too bloated and expensive to use, and this avisynth method produces combing artifacts and astronomically large files. I'd shoot native but the camera crashes if i so much as point it outdoors.

edit: mutabot's script works! Its the scripts at the beginning of the thread that dont.

[David Newman]

tonpole,

NeoScene will produces 24p output between 70-120Mb/s based on the quality setting you choose, using a high-quality I-frame wavelet. Long GOP AVCHD in the GH1 is 17Mb/s (unless you hack the firmware for a higher bit-rate.) So the data size increase is 4 to 7X, and is well worth it (in you hack the camera in increase is even less.) The datarate is equilivent to other I-frame intermediate codec, such as Avid's DNxHD or Apple's ProRES. With all intermediate codecs, the video is much easier to edit, another reason only $99 for NeoScene goes a long way to help tens of thousands of users.

[cbrandin]

David,

Does using the hack, which produces bigger MTS files, also produce bigger files out of NeoScene? I assume it does not (or, at least not much) because the NeoScene output is already much higher bitrates than AVCHD. Certainly if one choses all I frames it seems that it shouldn't make much difference.

Chris

[David Newman]

No, it will not produce noticabley larger files in NeoScene, in some cases you might get smaller files. The artifacts of AVCHD can produce false detail that CineForm has to compress. The datarate on the CineForm output is image dependent, a clean simple image will be lower data rate that a complex noisy one.

[cbrandin]

Thanks.

Chris

[mojo43]

Has anyone else tried mutabot (http://www.dvxuser.com/V6/member.php?u=59494)'s way of removing pulldown with success?

[tonpole]

Has anyone else tried mutabot (http://www.dvxuser.com/V6/member.php?u=59494)'s way of removing pulldown with success?


Yes, but it requires a few changes, so here's the updated script:



---- 24p.bat begin ----

for %%f IN (*.mts) DO (
echo directshowsource^("%%f"^).AssumeTFF^(^).SeparateFields^(^) > %%~nf.avs
echo Assert^(width^(^) == 1920, "Not a 1080p24 source"^) >> %%~nf.avs
echo x0 = selectevery^(10, 0,1^).Weave^(^) >> %%~nf.avs
echo x1 = selectevery^(10, 2,3^).Weave^(^) >> %%~nf.avs
echo x2 = selectevery^(10, 5,6^).Weave^(^) >> %%~nf.avs
echo x3 = selectevery^(10, 8,9^).Weave^(^) >> %%~nf.avs

echo Interleave^(x0,x1,x2,x3^) >> %%~nf.avs
)

for %%f IN (*.avs) DO (
echo VirtualDub.Open^("%%f"^); >> %%~nf.script
echo VirtualDub.video.SetCompression^("Lags",0,10000,0^); >> %%~nf.script
echo VirtualDub.video.SetOutputFormat^(11^); >> %%~nf.script
echo VirtualDub.SaveAVI^("%%~nf.avi"^); >> %%~nf.script
echo VirtualDub.Close^(^); >> %%~nf.script
)

for %%f IN (*.script) DO (
"C:\Program Files\VirtualDub\vdub.exe" /min /x /s %%~ff
)

for %%f IN (*.script) DO (
del /Q %%f
)

for %%f IN (*.avs) DO (
del /Q %%f
)

---- 24p.bat end --- A Few notes:

1. Make sure the vdub location is set correctly.

2. I set this to encode to lagarith so you can get lossless output at decent size. You can set this to whatever you want, just put the fourcc of whatever codec you want to use in the quotes, and this is case sensitive so try a few variations if it doesnt work. Also make sure your codec is set to output YUY2.

3. Put this in the same folder as your files and it will encode all of them automatically. Dont leave any spaces in any of the folder names in the hierarchy or it wont work.


//

[mojo43]

Fantastic!!! It looks great...Thanks a lot tonpole!

I can't see any artifacting, are you finding any problems with this way of removing pulldown?

[tonpole]

Ive compared it side by side to a locked scene using the native patch and its identical, no weird combing artifacts or chroma bleeds like with the other methods.

[mojo43]

Ya, I am not seeing anything either. I need to do some more tests, but all seems to look correct.

Here you go... a free pulldown removal method!! I slapped another loop in the batch script and did all of my folders on one go. Easy as pie!

Thanks...

[cbrandin]

This is a really nice find. This makes it possible to not have to worry about the Native 24p issue so much - which may not have a solution any time soon.

Nice work!

Chris

[ttatcs]

Does Tonpole's updated script also work for 25P (50i wrapper) source? if not , what shoud be revised? I wanna pure 25p source ready before it is imported into NLE .

[tonpole]

You'd have to ask the author but hes apparently hasnt been on in months. If it doesnt work you can always go over to doom9's avisynth forum and ask them to modify it for 25p.

[ttatcs]

I revised the script to output 25p for pal user.



---- 25p.bat begin
for %%f IN (*.mts) DO (
echo directshowsource^("%%f"^).AssumeTFF^(^).SeparateFields^(^) > %%~nf.avs
echo Assert^(width^(^) == 1920, "Not a 1080p25 source"^) >> %%~nf.avs
echo x0 = selectevery^(2, 0,1^).Weave^(^) >> %%~nf.avs

echo Interleave^(x0^) >> %%~nf.avs
)

for %%f IN (*.avs) DO (
echo VirtualDub.Open^("%%f"^); >> %%~nf.script
echo VirtualDub.video.SetCompression^("Lags",0,10000,0^); >> %%~nf.script
echo VirtualDub.video.SetOutputFormat^(11^); >> %%~nf.script
echo VirtualDub.SaveAVI^("%%~nf.avi"^); >> %%~nf.script
echo VirtualDub.Close^(^); >> %%~nf.script
)

for %%f IN (*.script) DO (
"C:\VirtualDub-1.9.8\vdub.exe" /min /x /s %%~ff
)

for %%f IN (*.script) DO (
del /Q %%f
)

for %%f IN (*.avs) DO (
del /Q %%f
)

---- 25p.bat end

Please help test it and correct me if I am wrong .

but , another problem arise : I put the oringinal mts file and pulldowned avi file into premiere cs3 and watched in the y/c waveform monitor. since the Lag codec considered to be lossless , the 2 files 's y/c waveform should be identical . but the result is NOT . the avi file has slightly narrower luminance width than the original mts file . I guess this is caused by 0-255 to 16-235 convertion . Just a guess . My question is "how to avoid this ?" . I want to losslessness convertion .

[mojo43]

Have you tried a different codec? I am converting to the canopus codec...

[ttatcs]

Have you tried a different codec? I am converting to the canopus codec...

you mean canopus hq ? I did not try it . I intend to try lossless codec . though I am getting realize that this lossless codec may not be a good choice for editing . yes , will try canopus later . But now I wanna know how to avoid this "lossless" loss .

[mojo43]

No, I meant Canopus lossless

[tonpole]

You might have accidentally converted to rgb, check your codec settings.

[mojo43]

Sticky???

[NPrincen]

Bumping this thread. Why are all the pay programs for pulldown removal stickied and not this free one? I hope the moderators decide to put this up as a sticky so people know there is an alternative.

[mojo43]

I asked but got no response....

[mojo43]

bump

[timmciglobal]

So I tried it and it turned a 2 minute long 220 meg .mts video into a 11 gigabyte mts.

Did I mess something up? Any ideas?

Tim

[mojo43]

Make sure that you set your compression settings in the script. Without any compression you will have very large file sizes. Or maybe you aren't compressing enough with your codec choice. Let me know if you need any more help...

[nathankw]

NOTE: Win 7 users must disable built-in mp4 decoder for this to work.


Doea anyone have any idea how to do that?
It seems to work OK for me on Win7 except that I'm still seeing some combing so I guess that's the build in decoder mucking things up?

[meinyvr]

thanks

i'm just a little confused about the whole 24p wrapper thing. What is the advantage of having it checked? is it only that you save time transcoding (sp) it from 60i to 24p....or does it change the quality of the colours etc? prevent jittery action?

Also do you do the pulldown before editing your masterpiece or after?

I use vegas video for example...can i do pulldown in that?

appreciate your patience

[mojo43]

thanks

i'm just a little confused about the whole 24p wrapper thing. What is the advantage of having it checked? is it only that you save time transcoding (sp) it from 60i to 24p....or does it change the quality of the colours etc? prevent jittery action?

Also do you do the pulldown before editing your masterpiece or after?

I use vegas video for example...can i do pulldown in that?

appreciate your patience

There is a difference. Checking the 24p means that you don't have to remove pulldown, but as I understand it is not stable. I remove pulldown and convert to a canopus codec and then edit. I don't believe that you can remove pulldown in vegas, but I could be wrong. I would just go and try it out and see if you can. Hope that helps!

[meinyvr]

There is a difference. Checking the 24p means that you don't have to remove pulldown, but as I understand it is not stable. I remove pulldown and convert to a canopus codec and then edit. I don't believe that you can remove pulldown in vegas, but I could be wrong. I would just go and try it out and see if you can. Hope that helps!

Thanks - yes that clears it up a bit.

[l.breyers]

Doea anyone have any idea how to do that?
It seems to work OK for me on Win7 except that I'm still seeing some combing so I guess that's the build in decoder mucking things up?

I have been using this script for a while and got awesome results, but recently i noticed combing and came back here to figure out what might be wrong, and saw that the author mentioned that win7 codecs had to be disabled for it to work. So i googled "disable win7 codecs" and found a program called win7DSfiltertweaker that disables all the builtin windows codecs, and tried the script and the combings gone! Unfortunately theres still a very strange problem with strobing, almost as if this script is throwing away frames.

[mojo43]

I have been using this script for a while and got awesome results, but recently i noticed combing and came back here to figure out what might be wrong, and saw that the author mentioned that win7 codecs had to be disabled for it to work. So i googled "disable win7 codecs" and found a program called win7DSfiltertweaker that disables all the builtin windows codecs, and tried the script and the combings gone! Unfortunately theres still a very strange problem with strobing, almost as if this script is throwing away frames.

Hmmm... I have never had any problems with it. Are you using the most recent version of the script?

[echohead]

For your convenience, I have packaged all of the necessary files into a ZIP archive that you can download here: http://www.filedropper.com/avchdworkflow
.

Thanks for the work, but no file shows up when I click on the link. Looking forward to checking this method out.

[l.breyers]

Hmmm... I have never had any problems with it. Are you using the most recent version of the script?

I looked at it and found that the reason strobing looked so bad was the window size i was viewing it at. I scaled the footage down to half its size and the strobing disappeared. Considering that its presence is how fast a feature tracks across the screen, bigger screen = bigger strobing.

[Pixelznbits]

This is the algorithm:

x0 = selectevery(10, 0,1).Weave()
x1 = selectevery(10, 2,3).Weave()
x2 = selectevery(10, 5,6).Weave()
x3 = selectevery(10, 8,9).Weave()



I am not a programmer but I understand this algorithm kinda but I have a question. What is keeping it from starting the pulldown pattern on the wrong frame and therefore throwing away the wrong frame? Is there something in the script that I don't recognize that is allowing it to start in the right place?

Also is this algorithm for removing 2-3-3-2 pulldown? I know this is a panasonic forum but for the canon hv30 that has standard 2-3 pulldown wouldn't I just modify the script so it throws away the fifth field in every group of 5?

[l.breyers]

I am not a programmer but I understand this algorithm kinda but I have a question. What is keeping it from starting the pulldown pattern on the wrong frame and therefore throwing away the wrong frame? Is there something in the script that I don't recognize that is allowing it to start in the right place?

Also is this algorithm for removing 2-3-3-2 pulldown? I know this is a panasonic forum but for the canon hv30 that has standard 2-3 pulldown wouldn't I just modify the script so it throws away the fifth field in every group of 5?

You could try asking over at the doom9 forums, they understand this stuff.

[PDR]

I am not a programmer but I understand this algorithm kinda but I have a question. What is keeping it from starting the pulldown pattern on the wrong frame and therefore throwing away the wrong frame? Is there something in the script that I don't recognize that is allowing it to start in the right place?

Also is this algorithm for removing 2-3-3-2 pulldown? I know this is a panasonic forum but for the canon hv30 that has standard 2-3 pulldown wouldn't I just modify the script so it throws away the fifth field in every group of 5?



A better approach is to use an adaptive algorithm like TIVTC. It finds field matches and decimates the duplicates, so it doesn't matter where you start in the cadence cycle

Download TIVTC from here
http://web.missouri.edu/~kes25c/

e.g
DirectShowSource("video.mts")
TFM()
TDecimate()

[jaksco]

Good stuff. Sticky?