Has anyone Filmedout GH1 Footage?

[Indyreel]

Indy... did you not read anything I wrote?

First of all the hack IS THE ONLY way to shoot native 1080/24p. With the standard firmware you are shooting 24p in 60i and need to spend money on a digital intermediate workflow for pulldown removal which ads an extra pain in the abung step. Secondly, the increased bitrate MASSIVELY effects picture quality.... doubles or triples the quality of the image in 1080p easily I would say.

Sorry Blackout,

I have been scatter brained lately and crazy busy trying to get the shots I need before the leaves change here in New England.

I am definately experiencing that macro blocking issue now, i.e. the exterior foliage shots, and realize that I need the patch to continue.

Could you please tell me the most straight forward way to do it?

Thank you

 

[Svart]

Indy... did you not read anything I wrote?

First of all the hack IS THE ONLY way to shoot native 1080/24p. With the standard firmware you are shooting 24p in 60i and need to spend money on a digital intermediate workflow for pulldown removal which ads an extra pain in the abung step. Secondly, the increased bitrate MASSIVELY effects picture quality.... doubles or triples the quality of the image in 1080p easily I would say.

Yeah but if you are already going through the step of transcoding into an intermediate codec for editing, then what's the big deal of pulldown removal? I have to transcode because I can't edit MTS files so I choose to use Cineform and in a single step process I get pulldown removal, 24 REAL frames, 4:2:2 colorspace and fast editing all in a couple of minutes. Nevermind the fact that the Native 24p output from the hack is full of strange errors like missing P frames and stuff. Why on earth would I want those errors in production footage? I'd rather go through the "hassle" and get clean and perfect footage.

I set up cineform to do a batch conversion on 2 days worth of clips, about 12 hours total footage time, and it took an hour or so to convert. I went and made lunch and came back and it was done. Big deal.

 

[Lpowell]

Yes, Cineform is high quality, but there are a few catches:

* $500 per workstation for NeoHD cause we need HD of course
* Lock-in with Firstlight since Apple Color is no go
* Cineform intra files are really huge, 5-10x AVCHD

That's what led me to 3-frame GOP AVCHD, though of course its only 720p30

 

[Svart]

Yes, Cineform is high quality, but there are a few catches:

* $500 per workstation for NeoHD cause we need HD of course

I edit HD and I just use Neoscene. 1920 x1080. That's HD isn't it..

* Lock-in with Firstlight since Apple Color is no go

Why? It's a MOV/AVI container file and the codec should plug into any editor just like any other codec would. I frequently trade files with an editor who uses macs while I use PCs. Haven't had a problem between the two of us coloring and editing..

* Cineform intra files are really huge, 5-10x AVCHD

As I said, AVCHD is not true 24 frames of information. (in the GH1) It's an I frame with a number of P frames defining the *changes* over time of the picture (GOP). Cineform and other intermediate codecs interpret the P frame changes and reconstruct REAL frames of data IN 4:2:2. This is why Cineform inflates the file size. Pretty much all intermediate codecs that recreate 24 frames from AVCHD(or any other compressed codec) will inflate file sizes. Those who don't create 4:2:2 files won't be as big.

You have to pay in size, to play in quality. I'm not sure why folks have such a hard time understanding this.

 

[Indyreel]

As I said, AVCHD is not true 24 frames of information. (in the GH1) It's an I frame with a number of P frames defining the *changes* over time of the picture (GOP). Cineform and other intermediate codecs interpret the P frame changes and reconstruct REAL frames of data IN 4:2:2.

Is DNxHD 145 one of those intermediate codecs that interpret the P frame changes and reconstruct REAL frames of data IN 4:2:2.

 

[Homunculus]

what? Svart how are you getting 4:2:2 are you talking about hacked mjpeg mode?
4:2:2 in 1080p acvhd mode? HUH?

 

[Svart]

what? Svart how are you getting 4:2:2 are you talking about hacked mjpeg mode?
4:2:2 in 1080p acvhd mode? HUH?

Yeah, cineform interpolates into 4:2:2 from 4:2:0

http://www.cineform.com/neoscene/specifications.php

DNxHD is a lossy codec but it's designed for multi-generation compression. You can select different encoding bitrates with it though: "DNxHD offers a choice of three user-selectable bit rates: 220 Mbit/s with a bit depth of 10 or 8 bits, and 145 or 36 Mbit/s with a bit depth of 8 bits."

Transcoding after doing pulldown removal should get your true 24 frames in most intermediates I believe. I might be wrong though.

DNxHD is very similar to JPEG.

ProRes422 is very similar to DNxHD, almost identical. Both of these are 4:2:2 as well. I assume these interpolate as well and don't do anything funky.

http://www.avid.com/US/solutions/workflow/DNxHD-Codec

http://en.wikipedia.org/wiki/ProRes_422

In reality they are all pretty similar in file sizes when encoded into similar bitrates. Most of these are meant to compete with each other. I just use cineform because I don't have a Mac and it works pretty well.

 

[Lpowell]

I edit HD and I just use Neoscene. 1920 x1080. That's HD isn't it..
It's a MOV/AVI container file and the codec should plug into any editor just like any other codec would. I frequently trade files with an editor who uses macs while I use PCs. Haven't had a problem between the two of us coloring and editing..

As I said, AVCHD is not true 24 frames of information. (in the GH1)

Yes, NeoScene does do HD. I was thinking 2K widescreen, which needs NeoHD.

The catch with Apple Color is it doesnt support Quicktime codecs, only ProRes. Not Cineform's fault but still an issue.

Actually, AVCHD does deliver 24 frames of info, P-frames are just as real as key frames, they're just compressed more efficiently. No intermediate codec can improve the quality of the original, at best it can only preserve it without visible degradation. Cineform's 422 color is only an advantage when you render comps back to the Cineform file. If you do your final render directly from the source AVCHD file, you wont need an intermediate file.

I'm not knocking it on its own terms - Cineform is good at what it does when thats what you need. Just pointing out the advantages to sticking with native AVCHD.

 

[Svart]

Yes, NeoScene does do HD. I was thinking 2K widescreen, which needs NeoHD.

The catch with Apple Color is it doesnt support Quicktime codecs, only ProRes. Not Cineform's fault but still an issue.

Actually, AVCHD does deliver 24 frames of info, P-frames are just as real as key frames, they're just compressed more efficiently. No intermediate codec can improve the quality of the original, at best it can only preserve it without visible degradation. Cineform's 422 color is only an advantage when you render comps back to the Cineform file. If you do your final render directly from the source AVCHD file, you wont need an intermediate file.

I'm not knocking it on its own terms - Cineform is good at what it does when thats what you need. Just pointing out the advantages to sticking with native AVCHD.

Unfortunately this:

P-frames are just as real as key frames, they're just compressed more efficiently

Is not true.

A "P" frame, short for predictive frame, only contains the change vectors from the preceding I frame in the GOP.

That means they only contain partial data showing only the data that changes and cannot be used on their own. Anything in the I frame that does not move, is not included in the P frame at all.

They are not compressed "more efficiently" at all either. The compression is based on the compressing processor interpreting the changes between I frames and creating a number of P frames based on the GOP length. The lack of the full data of the I frame IS how the codec keeps it's small file size.

The AVCHD codec *represents* 24 frames of *information* but it's not 24 true I frames. Only a dedicated decompressor can interpolate the changes made between P frames, analyze the I frame and then reconstruct a true I frame from this.

The Native 24p is only outputting a *representation* of a 24fps file because it's still in compressed form, see above.

However, this is all moot really since the GH1 Native 24p hack has serious issues that render the codec pretty much invalid with it's periodic double I frames and missing P frames with some of those being of odd size and/or full of random bits. It *may* look OK when watching but every single time someone runs an analysis on a Native 24p file, it shows crap. I'd still rather just do pulldown removal than have hidden file errors.

 

[Blackout]

Yeah but if you are already going through the step of transcoding into an intermediate codec for editing, then what's the big deal of pulldown removal? I have to transcode because I can't edit MTS files so I choose to use Cineform and in a single step process I get pulldown removal, 24 REAL frames, 4:2:2 colorspace and fast editing all in a couple of minutes. Nevermind the fact that the Native 24p output from the hack is full of strange errors like missing P frames and stuff. Why on earth would I want those errors in production footage? I'd rather go through the "hassle" and get clean and perfect footage.

I set up cineform to do a batch conversion on 2 days worth of clips, about 12 hours total footage time, and it took an hour or so to convert. I went and made lunch and came back and it was done. Big deal.

Because the 24p native footage is better. That's why. It always outputs visually better than going to an intermediate. I have cineform, but rarely use it now unless, again, I have some big green screen or color correction shoot.

 

[Svart]

I get what you are saying, but a file that has missing parts and double frames and other random bits is just useless to me regardless. I'd rather have some known and repeatable data loss through a compression technique than some unknown data loss that is totally random at best.

At least with an intermediate you still have your original files to use if something goes amiss. With the Native 24p your original files are the ones that are screwy and there isn't a thing you can do about it. Just because you *can* use something doesn't mean that you *should*.

Besides, in my old side by side testing before finding the Native 24p crash and swearing it off, I couldn't tell any difference. Seriously. I've even revisited it and found that I couldn't tell the difference at all.

But use whatever makes you happy. I just can't knowingly give someone a file that I know has bugs in it. Maybe it's just my perfectionism talking.

 

[Lpowell]

A "P" frame, short for predictive frame, only contains the change vectors from the preceding I frame in the GOP.

Sorry, but you're mistaken about how P-frames work in AVC codecs. A good summary on its compression techniques is available here:

http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Features

One key point is that a P-frame can reference back to the decoded image details of the immediately preceding frame, regardless of whether it was encoded as an I-frame or a P-frame. When there's relatively little motion in a video, the changes can be compressed much more efficiently than with an I-frame-only codec.

That means they only contain partial data showing only the data that changes and cannot be used on their own. Anything in the I frame that does not move, is not included in the P frame at all.

Exactly, and that's why it's such a powerful compression technique. Instead of redundantly re-encoding details that were captured in a previous frame, each P-frame contains only the data needed to reconstruct the changed parts of the image. (Assuming an adequate amount of bitrate, of course.)

The AVCHD codec *represents* 24 frames of *information* but it's not 24 true I frames. Only a dedicated decompressor can interpolate the changes made between P frames, analyze the I frame and then reconstruct a true I frame from this.

Nope, there's nothing "untrue" about the data contained in a P-frame. The decoded P-frames are not approximated or interpolated, they are reconstructed from data that is no less precise than what is contained in I-frames. The only difference is that I-frames are encoded from scratch while P-frames are based on decoded reference frames that precede them.

However, I do agree with your reasons for avoiding Native 24p mode at this time, as it is still unreliable and has been shown to produce corrupted frames. Unlike the Native 24p non-interlaced 1080p mode, neither the progressive 720p30 nor the interlaced 1080p24 AVCHD modes I developed in my Fast Action 3-Frame GOP Patch are afflicted by these problems:

http://dvxuser.com/V6/showthread.php?t=222791

 

[Homunculus]

Yeah, cineform interpolates into 4:2:2 from 4:2:0

http://www.cineform.com/neoscene/specifications.php

DNxHD is a lossy codec but it's designed for multi-generation compression. You can select different encoding bitrates with it though: "DNxHD offers a choice of three user-selectable bit rates: 220 Mbit/s with a bit depth of 10 or 8 bits, and 145 or 36 Mbit/s with a bit depth of 8 bits."

Transcoding after doing pulldown removal should get your true 24 frames in most intermediates I believe. I might be wrong though.

DNxHD is very similar to JPEG.

ProRes422 is very similar to DNxHD, almost identical. Both of these are 4:2:2 as well. I assume these interpolate as well and don't do anything funky.

http://www.avid.com/US/solutions/workflow/DNxHD-Codec

http://en.wikipedia.org/wiki/ProRes_422

In reality they are all pretty similar in file sizes when encoded into similar bitrates. Most of these are meant to compete with each other. I just use cineform because I don't have a Mac and it works pretty well.

"You're not talkin' my language little fella."

listen if the bit depth simply is not there from the output, it is 8bit, how can you possibly get some codec to transcode it to 4:2:2 out of 4:2:0. Is it a 'fake' sort of thing sort of like uprezzing standard def footage to HD or what?

Can you post footage of yours with this 1080p avchd 4:2:2 .........?

 

[Lpowell]

...if the bit depth simply is not there from the output, it is 8bit, how can you possibly get some codec to transcode it to 4:2:2 out of 4:2:0. Is it a 'fake' sort of thing sort of like uprezzing standard def footage to HD or what?

Your suspicions are correct - the original 4:2:0 data cannot be improved by transcoding it into 4:2:2, but that increased color depth may help prevent visible degradation during post-production.

When you color grade original or transcoded footage, the modified image data is maintained in full precision as long as it remains in memory. When you write it back to a file, however, the color graded images will be truncated to the color depth supported by the file format. 4:2:2 file formats store inherently higher precision data than 4:2:0 formats, hence the potential advantage to transcoding up from 4:2:0 to 4:2:2.

 

[Svart]

Sorry, but you're mistaken about how P-frames work in AVC codecs. A good summary on its compression techniques is available here:

http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Features

One key point is that a P-frame can reference back to the decoded image details of the immediately preceding frame, regardless of whether it was encoded as an I-frame or a P-frame. When there's relatively little motion in a video, the changes can be compressed much more efficiently than with an I-frame-only codec.


Exactly, and that's why it's such a powerful compression technique. Instead of redundantly re-encoding details that were captured in a previous frame, each P-frame contains only the data needed to reconstruct the changed parts of the image. (Assuming an adequate amount of bitrate, of course.)


Nope, there's nothing "untrue" about the data contained in a P-frame. The decoded P-frames are not approximated or interpolated, they are reconstructed from data that is no less precise than what is contained in I-frames. The only difference is that I-frames are encoded from scratch while P-frames are based on decoded reference frames that precede them.

However, I do agree with your reasons for avoiding Native 24p mode at this time, as it is still unreliable and has been shown to produce corrupted frames. Unlike the Native 24p non-interlaced 1080p mode, neither the progressive 720p30 nor the interlaced 1080p24 AVCHD modes I developed in my Fast Action 3-Frame GOP Patch are afflicted by these problems:

http://dvxuser.com/V6/showthread.php?t=222791

Unfortunately I'm not mistaken. I've read that article many times. That article explains, in more detail, exactly what I said. Maybe I simplified it too much to be of use? Here's the explanation from a professional datasheet:

Short for predictive frame, or predicted frame, a video compression method used by the MPEG standard. In a motion sequence, individual frames of pictures are grouped together (called a group of pictures, or GOP) and played back so that the viewer registers the video’s spatial motion. P-frames follow I-frames and contain only the data that have changed from the preceding I-frame (such as color or content changes). Because of this, P-frames depend on the I-frames to fill in most of the data. P-frames and B-frames are also referred to as delta frames.

maybe this can enlighten you?

http://www.cs.cf.ac.uk/Dave/Multimedia/node249.html

Again, the P frames ONLY CONTAIN movement vectors, they do NOT contain any visual information like an I frame. You are correct that a P frame can be referenced back to the preceding frame, because it was predicted from that frame.

What you are saying is more like MJPEG where each frame is compressed and stored. In AVCHD, the I frame is compressed and then a P frame is predicted based on movement from previous frames. Only the moving pixels/macroblocks are stored in a P frame and then another frame of movement is stored in the next P frame, and so on, until you reach the end of the GOP. The process then starts over with another I frame.

the changes can be compressed much more efficiently than with an I-frame-only codec.

The changes aren't "compressed" at all. That is fact. during large movement, you have lots of changes and therefor larger bitrate spikes(as you know) and then when there is little movement, there is little to be stored in the P frame and very little bitrate change(as you also know). I think you may be confusing "compression" with "size" in this case.

 

[Svart]

"You're not talkin' my language little fella."

listen if the bit depth simply is not there from the output, it is 8bit, how can you possibly get some codec to transcode it to 4:2:2 out of 4:2:0. Is it a 'fake' sort of thing sort of like uprezzing standard def footage to HD or what?

Can you post footage of yours with this 1080p avchd 4:2:2 .........?

Any 4:2:2 intermediate that encodes a 4:2:0 source will be interpolating. That means that it will base it's assignment of bit depth on a "best guess" type of scenario. It's not fake, but it's not creating detail where there was none either. It's simply subdividing the data that is already there into finer detail.

 

[Homunculus]

Any 4:2:2 intermediate that encodes a 4:2:0 source will be interpolating. That means that it will base it's assignment of bit depth on a "best guess" type of scenario. It's not fake, but it's not creating detail where there was none either. It's simply subdividing the data that is already there into finer detail.

I think I understand that. That still allows for much higher quality post grading etc so I guess tha'ts good .

 

[Svart]

I think I understand that. That still allows for much higher quality post grading etc so I guess tha'ts good .

It's only going to affect precision. overall quality(detail, etc) is the same. The color filters are just able to be more precise.

You probably won't notice the difference unless you do very heavy grading though but with heavy grading you'll run into other problems anyway.

 

[Indyreel]

Yeah, cineform interpolates into 4:2:2 from 4:2:0

http://www.cineform.com/neoscene/specifications.php

DNxHD is a lossy codec but it's designed for multi-generation compression...

Transcoding after doing pulldown removal should get your true 24 frames in most intermediates I believe.

Svart,

What do you mean by "DNxHD is a lossy codec" ?

Also, I keep hearing about pulldown removal before transcoding. I'll be honest, I'm confused, as I have simply been transcoding the AVCHD file in Avid and it plays fine on the timeline.

Am I missing a step here (i.e. should I be doing pulldown removal first)?

 

[Svart]

Well most intermediates are lossy, which means that when they recompress they lose a small amount of the original information. There are such things as lossless compression codecs but they can never match the reduction in size of a lossy codec and in my experience don't edit very well.

when the GH1 creates an FHD AVCHD file it's 24p that is interlaced into a 60i stream. This is called Telecine/pulldown:

http://en.wikipedia.org/wiki/Telecine

To get the original 24p out of the 60i container is a process called Reverse Telecine or simply pulldown removal. I'll let the wiki article explain it.

Most TV's and modern DVD players can remove telecine on the fly, play 24p directly or simply play the 24p-in-60i container these days. The telecine process was originally developed because older TVs and such could not play 24p, only 60i.

I think most modern NLEs can edit 60i wrapper stuff as easily as they can 24p so it's really a matter of taste although some claim that editing 24p footage makes a cleaner render. I don't know if it does or not. I've always edited 24p so it's what I know.