Clever, FREE pulldown-removal & transcoding workflow via AviSynth

ydgmdlu

ydgmdlu

Veteran
Inspired by the work of SLoNiCK, I've developed a primarily Windows-based workflow that's completely free (and legal) and offers at least as good, if not better, results than anything else attempted so far. Although the first-time set-up is a minor hassle, I think that processing individual files will be less work than the other methods that have been proposed and tested (other than, of course, CineForm Neo Scene). There is a slightly convoluted way of getting this to work with Mac OS X, which I will explain later. We're using Avid DNxHD as the intermediate codec. It's comparable to ProRes; in fact, Apple created ProRes for FCP to compete with Avid's codecs for its NLEs.

What you need:


For your convenience, I have packaged all of the necessary files into a ZIP archive that you can download here: http://www.filedropper.com/avchdworkflow

Set-up instructions:

  1. Install AviSynth.
  2. Copy TIVTC.dll and QTSource.dll to C:\Program Files\AviSynth 2.5\plugins
  3. Install Haali Media Splitter. During installation, make sure that "Enable MPEG-TS support" is checked.
  4. Install ffdshow tryouts. During installation, make sure that ffdshow is set to decode H.264 video.
  5. Install Avid QuickTime codecs.

To remove pulldown and transcode:

  1. Open script.avs in Notepad.
  2. Change what's between the quotation marks in the first line to match the name of the file that you want to process.
  3. Change what's between the quotation marks in the last line to whatever you want to name the output file.
  4. Open script.avs in Windows Media Player. WMP will tell you that it doesn't recognize the type of file, but tell it to try to play anyway. Check the box so that you won't get the same warning every time.
  5. A dialog box will soon open asking you to specify the QuickTime encoding configuration. Just change the compression type to "Avid DNxHD Codec," depth to "Best Depth," and resolution to one of the 1080p/23.976 options (your choice). Click OK.

The dialog box will close, and WMP will slowly play the video. Do not interrupt the playback. Once it reaches the end, DO NOT close WMP. The QuickTime encoder will still be encoding the output file, and it NEEDS the input file to be open while it's working. Unfortunately, there's no progress bar of any sort. However, you can monitor the progress by opening the destination folder in a Windows Explorer window. Set the view mode of that window to "Details." If you keep pressing the F5 key, you'll notice that the size of the output file will be rapidly increasing. Eventually, that number will stop growing, at which point the process is complete. You can play the video in QuickTime to review the result.

If you install the Mac versions of the Avid codecs, you can then edit the transcoded footage in FCP.

Right now, the directions above are a proof-of-concept. With the process, I was able to successfully transcode SonicStates's cat clip (the example.MTS in the package) without any artifacts. There are ways to automate the process a little more; for example, SLoNiCK showed how you could write a batch script to generate AviSynth scripts for all of the MTS files in a given folder. I imagine that the entire process could be automated if a programmer could write a front-end that handles all of the steps. I don't think that it would be very hard, but I'm not a Windows programmer myself.

If you don't have easy access to a Windows computer, then there is a way to do what I described. There's a free application called VirtualBox that would allow you to create a virtual Windows XP PC inside of Mac OS X. It's a lot like Parallels or VMware Fusion, except that it's free. All that you would need is to "borrow" a copy of Windows XP to install, and you still wouldn't have to pay anything. The virtual machine would be set-up to share a folder with the host Mac computer, and you would use that shared folder as your transcoding destination. In other words, transcode in Windows and instantly switch back to Mac to bring the files into FCP. This would actually be a viable workaround for any workflow that demands Windows, such as the current version of Neo Scene.

Let me know if you have any questions, and please give me feedback on this.
 
Hey thanks for putting this together. I'm not really using windows anymore but would love to hear how this works for people. Virtualbox sounds like an interesting workaround and I'll check it out.
 
I have set up a similar workflow for CS3 but without the pulldown bit, because I am in PAL land.

All I needed to install was Avisynth, PremiereAVSplugin, Haali Media Splitter and ffdshow. Then I use this batch script to generate avisynth scripts for all MTS-files in a given folder:

set pfs_path=.\
for /f "tokens=*" %%a in ('dir/b/s ^"%pfs_path%\*.mts^"') do echo directshowsource("%%a") > %%a.avs
exit

Then I just edit the avs-files. They play kind of slow so I installed the trial version of MPEG Pro. With that I can edit the original MTS-files with ease and then replace them with the avisynth scripts before final output.

I just wanted to confirm that this method works great, and it helped me in my decision to keep waiting for the GH1.
 
Last edited:
I recommend encoding with a different intermediate codec, in that case. But you could also try not using an intermediate codec at all and just importing the AVS script into Edius. But if you do that, you must delete the last line from the script, so that it doesn't attempt to output a QuickTime when you open it!
 
I downloaded the example. I'm still seeing a chroma error when scrubbing through. Six frames in around the eyes. See it?

Though most people are not going to notice it at all.
 
Yes, I do see it when I'm going frame-by-frame. It's not too bad, but I agree that Cineform is better (which is to say, smoother). Personally, I'd buy Cineform and use it instead, as it's MUCH easier, faster, and better compressed. My workflow is designed for people who are too cheap for it. :-)

I'm going to post a video, soon, demonstrating how gloriously smooth the VirtualBox workaround is, in case nobody wants to wait for Cineform to fix their Mac version. Stay tuned.
 
ydgmdlu

Why did you choose TIVTC? Looks like it makes some kind of smart adaptive deinterlacing and then removes duplicated frames.

Polanski

This works great, though I can't use the quicktime in Edius... It shows nothing in Edius...

Remove this line from ydgmdlu's script:
QTOutput("example.mov")
as long as you are not interested in .mov, save script, drop into VirtualDub (it's free - http://www.virtualdub.org/ ), go to "Video - Compression" menu (or press ctrl+p), select codec (CanopusHQ), go to "File - Save as AVI" (or press F7) and save the .avi. The result will fit Edius quite well.
 
Last edited:
SLoNiCK said:
ydgmdlu

Why did you choose TIVTC? Looks like it makes some kind of smart adaptive deinterlacing and then removes duplicated frames.

Polanski

This works great, though I can't use the quicktime in Edius... It shows nothing in Edius...

Remove this line from ydgmdlu's script:
QTOutput("example.mov")
as long as you are not interested in .mov, save script, drop into VirtualDub (it's free - http://www.virtualdub.org/ ), go to "Video - Compression" menu (or press ctrl+p), select codec (CanopusHQ), go to "File - Save as AVI" (or press F7) and save the .avi. The result will fit Edius quite well.
Click to expand...
TIVTC is not a smart adaptive deinterlacer. It's a comprehensive inverse telecine solution. It finds the duplicated fields generated by pulldown and properly reconstructs the original 24p (or 25p) signal. It only uses smart adaptive deinterlacing when it finds material that wasn't telecined (i.e. native NTSC video).

TIVTC is actually reputed to be the best inverse telecine filter for AviSynth. It'll work far better than your script, I think, since the filters that you use are blind to whatever pulldown pattern the GH1 actually employs. In other words, your script might actually weave and select the wrong frames. Just think about it.
 
ydgmdlu

The pulldown removal algorithm itself is very simple and periodic (applied to every other 5 frames). You can write it in basic AviSynth functions in less than dozen strings (so I'm a little bit confused about some 'special proper pulldown removal' that everyone is looking here). So the only thing that any 'automatic pulldown remover' has to do is to guess the period of pulldown. If it misses then you'll get still very interlaced video on output. It's hard not to notice it. But if it hits then you'll get quite nice progressive result without any sight of interlacing (at least on such clean video as that kitty). So I've got curios why there are still some slightly noticeable interlacing artifacts after TIVTC.

And heading back to GH1 video... I've tried my script with about 20 clips from different sources, and they all started with the same pulldown pattern. So it's safe to use with raw video from camera. More to say, if you cut clips on I-frames the pulldown pattern won't change. The only case when this script can fail is processing of video that was randomly cut. For example when someone edits GH1 footage in 60i project, outputs it to single file and wants to get 24p out of it.
 
If your script really works 100% of the time, then I would recommend it over mine, since it would seem to process the video a lot faster. But I'm guessing that even then, it's no match for CineForm or perhaps even VoltaicHD. But again, our AviSynth solutions are designed to be free...
 
Update for you on CineForm pulldown removal. The Mac version of Neo Scene (and HD/4K) now support a fix for the GH1 pulldown, so work just like the PC version. We hope we've priced this tool low enough to make it the obvious choice or Mac or PC.
 
ydgmdlu: the link http://www.filedropper.com/avchdworkflow does not seems to work

the avisynth code for GH1 pulldown removal is just:

Telecide(guide=1,post=2)
Decimate()

this outputs a 23.976 perfectly pulldown removed video to virtualdud . Or you can even load the script in premiere to edit the video directly avoiding the virtualdub saving-to-disck step.

But, if you preffer to save the video for editing, better do it with a lossless codec, try these free codecs:

http://lags.leetcode.net/codec.html
http://neuron2.net/www.math.berkeley.edu/benrg/huffyuv.html


not as good as cineform codec, cause the resulting files are bigger, but apart from that are excellent codecs!
 
Last edited:
Also, comparing the Neoscene result to the AVISynth result, the Cineform just looks a bit sharper. I'm not sure what kind of fairy dust they are sprinkling in over there at Cineform, but Neoscene offers a better image after pulldown than both AVISynth and TMPGENC Express(although TMPGENC is very close).
 
DrBlaz said:
ydgmdlu: the link http://www.filedropper.com/avchdworkflow does not seems to work
Click to expand...
Thanks for letting me know. I guess FileDropper deleted the file after people stopped downloading it. I'll re-upload it soon.

the avisynth code for GH1 pulldown removal is just:

Telecide(guide=1,post=2)
Decimate()

this outputs a 23.976 perfectly pulldown removed video to virtualdud . Or you can even load the script in premiere to edit the video directly avoiding the virtualdub saving-to-disck step.
Click to expand...
Basically, it works the exact same thing that my script does. But having choices is nice.

But, if you preffer to save the video for editing, better do it with a lossless codec, try these free codecs:

http://lags.leetcode.net/codec.html
http://neuron2.net/www.math.berkeley.edu/benrg/huffyuv.html
Click to expand...
I highly doubt that people here want to work with losslessly compressed files. They are simply too large. That's why I didn't recommend a lossless codec in my package.
 
John Caballero said:
Do the 24p pulldown files from Neo Scene work with Edius 5?
Click to expand...

I have to admit I never tried Edius. Neo Scene uses standard codec APIs like Video for Windows and DirectShow, if Edius uses either of those it should work. With the free trial you will find out pretty quickly.
 
John Caballero

Do the 24p pulldown files from Neo Scene work with Edius 5?

It accepts this clip on timeline, but looses framebuffer on Q6600@3200 with 85% CPU load. I've tried different quality setting and different codecs for Edius project, but didn't succeed.

Schmoe

Also, comparing the Neoscene result to the AVISynth result, the Cineform just looks a bit sharper.

I've noticed only the difference in red channel level. Left - avs, right - neo.
attachment.php


ydgmdlu
DrBlaz

According to mans, both Telecide(guide=1,post=2) and TFM(order=-1, mode=5, PP=7) make some adaptive deinterlacing thus showing some artifacts.

I tried to create script to a) throw out interlaced frames after DoubleWeave, b) to remove dublicated frames. Here what I made:

directshowsource("_your_clip_.MTS")
AssumeTFF().doubleweave()

x01 = selectevery(10,0)
x02 = selectevery(10,1)
x03 = selectevery(10,2)
x04 = selectevery(10,3)
x05 = selectevery(10,4)
x06 = selectevery(10,5)
x07 = selectevery(10,6)
x08 = selectevery(10,7)
x09 = selectevery(10,8)
x10 = selectevery(10,9)

CmbTrsh = 2

y01 = ConditionalFilter (x01, x02, x01, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y02 = ConditionalFilter (x02, x03, x02, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y03 = ConditionalFilter (x03, x04, x03, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y04 = ConditionalFilter (x04, x05, x04, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y05 = ConditionalFilter (x05, x06, x05, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y06 = ConditionalFilter (x06, x07, x06, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y07 = ConditionalFilter (x07, x08, x07, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y08 = ConditionalFilter (x08, x09, x08, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y09 = ConditionalFilter (x09, x10, x09, "IsCombedTIVTC(CmbTrsh)", "equals", "true")
y10 = ConditionalFilter (x10, x09, x10, "IsCombedTIVTC(CmbTrsh)", "equals", "true")

Interleave(y01,y02,y03,y04,y05,y06,y07,y08,y09,y10)
TDecimate(mode=0,cycle=10)
TDecimate(mode=0,cycle=9)
TDecimate(mode=0,cycle=8)
TDecimate(mode=0,cycle=7)
TDecimate(mode=0,cycle=6)
TDecimate(mode=0,cycle=5)

It uses TIVTC for guessing interlaced frames and dups removal. Note "CmbTrsh = 2". If you're getting some wrong result then try to play with it. It is interlaced frames detection threshold level, range 0-255. It can go mad on noisy video with such low level.
 

Attachments

  • avs_neo_pr.jpg
    avs_neo_pr.jpg
    323.3 KB · Views: 0
The color difference are likely due to a YUV to RGB color conversion using 601 rather Rec.709 color math. Neo Scene's conversion remain in the source YUV color space, no RGB conversions are used, so Neo is likely the correct output. We don't avoid RGB just for color and quality reasons, we find it is a faster also.
 
You must log in or register to reply here.
Back
Top