Advice on how to remove letterboxing?!! I like pickles.

Ferret Lady

Ferret Lady

Well-known member
Hi guys. need some advice here before I go messing around with it myself... I had a 4 camera shoot the other night. i instructed each camera to use 16:9 aspect ratio. two of these cameras were xh a1's, and they did what they were supposed to. the other two cameras were a dvx 100 and an hvx 200. i believe they selected 'letterbox', because although the aspect ratio seems to look correct, it's got the black bars on the top and bottom. please see picture below:

letterbox.jpg



so here's my issue. i've checked off anamorphic for the sequence, as well as all 4 clips. but as you can see by the pic above, this is not gonna match up. please, does anyone have any suggestions on what to do about this issue? the only thing I can think of is for the letterbox ones, to scale the image up until the black lines are gone. but i know i'm gonna lose quality if i do that and don't want that unless it's my only option.

btw, should they have selected 'squeeze' instead of letterbox?

Thanks in advance if anyone can give me some tips here!!!
 
Oh ya, i tried one thing. instead of trying to remove letterboxing on the clip on the right, i changed the aspect ratio of the clip on the left to -27.5. this is the number i found got the black lines to match up closely enough but now i'm squishing the footage on the left which is already 16:9 so that can't be right...

aspect.jpg
 
use compressor to match the cameras formats/ratios.
you be cropping out some of the picture, but there is no way around it.

scaling in FCP is a quick fix but the quality loss is alot more noticeable.

hope that helps!
shawn
 
ya i have quicktime pro. so you're saying i should export the clips into quicktime, and then somehow in there i can get rid of the letterboxing? how?
 
you should be able to resize and crop out the letterboxing.

doing this conversion before you edit will yield a higher quality.
(rather than scaling in FCP-which really sucks)

sorry i dont have the exact specs
 
couldnt find a way to crop in quicktime pro.

and now i have compressor. but your suggestion was to crop the OTHER footage so that it has black lines too? i really don't want to do that...
 
They definitely should have chosen squeeze mode. Choosing letterbox means you are just placing a black matte over the footage in-camera - and presto, resolution loss.

If you don't want to mess with the A1 footage (left side in your grabs), you will have to scale up the other footage. To see how it will look, turn off anamorphic on the letterboxed footage and scale it up until the matte is gone. You should see that everything is proportional and the shots will match better.

However, Shawn is right, scaling in FCP sucks. His advice is probably better than mine on what to do about that - I have never used Quicktime or Compressor to do this.
 
So, it's a slow day at work and I offered to fool around with some Compressor settings to see if I could make this work. Here's what I did, and I am pretty pleased with the result:

- Open the clip in FCP, place in the timeline, and then File>Export>Using Compressor

- Upon entering Compressor, the first thing I do is open the Settings tab and click the + icon to create a new setting. I choose "QuickTime Movie."

- Give it a name and description, save it. Then, in the "Encoder" tab, click on the "Settings" button next to Video.

- For Compression Type, choose DV/DVCPro - NTSC (or whatever your footage orginated in - no need to do uncompressed or transcode here).

- Frame Rate = Current

- Compressor Quality = Best

- Scan mode = however you shot it (in this case, I chose progressive)

- Aspect Ratio = 16:9 **this is important, as you want Compressor to do the interpolation for you, and you will be working in a 16:9 frame when you return to your edit

- Click Ok.

- Fix your Audio "Settings" so that they match the originating footage - again, no need to transcode here. Will only cause headaches later.

- Skip over to the "Geometry" tab - click the icon second from the right. It has a square overlayed with a circle and triangle (how clever!)

- Change "Crop to:" to Custom

- Enter 50 for the top, 52 for the bottom ***This worked for Alison's footage, your mileage may vary***

- Make sure that "Pixel Aspect" says NTSC CCIR 601/DV (16:9)

- Save your settings again.

- Apply these settings to your clip, choose your destination and submit it. When processing is complete, you should be able to open the new clip up in a 16:9 timeline and it will look pretty damn good. Much better than trying to do it in FCP.

Good luck.
 
Last edited:
ok, i'm gonna try this now. i've never used compressor before but your directions are very good so i should be ok. i will let you know as soon as i'm done if it works or i need help. thank you so much Will. if this works, your a life saver!
 
Will Clegg said:
They definitely should have chosen squeeze mode. Choosing letterbox means you are just placing a black matte over the footage in-camera - and presto, resolution loss.
Click to expand...

I know you got it done already but I would be interested in knowing what version of Final Cut you are using? I find the "squeeze mode" with the DVX to be a step down from the 4:3 letterbox. The edges just never look as sharp as they should be. When I run into a situation where I must use true 16:9 and 4:3 LB in the same sequence I create a 4:3 sequence and edit away. The anamorphic just doesn't have black so Final Cut fill it in. If I need a 16:9 output I just crop the output. This is much quicker then transcoding all the footage after capture or scaling.
 
Not sure what version Alison was using - I was using FCP 6.0.4, Compressor 3.0.3.

The trade off is between vertical and horizontal resolution - in squeeze mode you are using a different pixel aspect ratio, and in 4:3 LB you are actually using fewer lines.

Here's an example of something I shot a long while ago using squeeze mode on the DVX: http://vimeo.com/1215461

Even when it is scaled up to 720p (which it is for that version on vimeo), I still think it looks pretty sharp. I would rather stretch those pixels out slightly with the squeeze mode than lose vertical resolution by choosing LB. But neither is really a great option.
 
You must log in or register to reply here.
Back
Top