Possible breakthrough on Native 24p!

[cbrandin]

I've been digging into the GH1 firmware code and found that overall bitrate appears to have a big effect on the maximum speed that flash memory writes can occur. I’ve always wondered why the write speed error comes up in AVCHD mode when clearly the data rates are well below what the card can handle. I think I found the answer.

When the write speed error comes up it’s because the calculated maximum write rate has been exceeded – whether the card can handle more or not. I tried the Lpowell settings but changed two things: I doubled the overall bitrate to 86,000,000 and I checked Native 24p. Upping the overall bitrate nearly doubled the calculated flash write rate (i.e. the fastest write rate the camera attempts to write to flash). It appears to be working! I’m not getting any lockups and the clips look great. I tested this with highly detailed scenes, fast shutter speeds, everything I normally do to break the codec and it seems to be standing up.

We probably need to do some fine tuning and more testing (hint, hint, Lpowell), but so far it’s looking pretty good. I ran my tests with a Transcend class 10 card.

Chris

[Vitaliy Kiselev]

May be you are right here. :-)
It can be good news, as native is much more usable :-)

[butler360]

Eagerly subscribing to this thread! :2vrolijk_08:

[x_WOrPIG_x]

Lol. Right when I thought I had a pretty good setting , I found this thread. I'll test this out as well.

[cbrandin]

The theory behind this is that Native 1080 24p frames are twice as large as 720p and wrapped 1080p frames. The overall bitrate affects the instantaneous maximum write speed. With frames twice as large I surmised that the instantaneous write speed would have to be correspondingly faster. The video bitrate keeps the average bitrate over time to a manageable level.

Lets hope this holds up.

Chris

[Vitaliy Kiselev]

My latest theory was that it is not bitrate at all, but rather time related constant (being multiplier of 90000 ticks) :-)

May be it is good to look at all 18.000.000 locations and check routines logic carefully.

[cbrandin]

Yeah, that makes sense to me. At first I thought it was buffering too, but now I agree, it has more to do with transferr rate and write frequency. Youll notice that the pre-calculated minimum interval between writes is much smaller (the first four bytes of each 192 byte packet are the internal time stamp and the difference between them is the write interval) with the overall bitrate set high. The interval between packets is always a multiple of this pre-calculated number. Thus, the minimum interval absolutely limits the maximum frequency that packets can be written.

Chris

[cbrandin]

There are still a few things about the Native 24p streams that are a little strange - like an extra P frame every now and then and an empty frame also. It doesn't seem to affect how the clips look, though, so I'm not sure what to make of it. Also, I've seen some of this with 720p clips. It wouldn't be visible with wrapped clips because of the pulldown, etc... We might just be seeing artifacts that always existed and aren't related to the hack.

Chris

[Vitaliy Kiselev]

It is very interesting thoughts.

[Vitaliy Kiselev]

There are still a few things about the Native 24p streams that are a little strange - like an extra P frame every now and then and an empty frame also. It doesn't seem to affect how the clips look, though, so I'm not sure what to make of it. Also, I've seen some of this with 720p clips. It wouldn't be visible with wrapped clips because of the pulldown, etc... We might just be seeing artifacts that always existed and aren't related to the hack.
Chris

Most probably this strange things happen in original code.
First, this is due to architecture used (very hard to test).
Second, routines have so many hardcoded things, I am amazed.

[cbrandin]

Look at the number shown in the attached Stream Parser screenshot. It represents the theoretical maximum speed the flash memory will be written. My Transcend class 10 card can do about 15MBs, and this value is somewhat lower than that. If my theory is correct then we would want to set the Overall Bitrate so this number equals just below the flash card maximum write speed. That way the camera will spend the minimum time possible servicing the flash memory write routine, and have processing resources available for other things.

Who knows, once that has been set to the maximum, the other settings may be able to be re-thought. It would be interesting to set this with a SanDisk class 10 card so that the transfer rate comes to 160Mbs (equivalent to 20MBs) - maybe even higher, and then see how high the Video Bitrate can go before instability appears. My thinking now is that the Overall bitrate is related mostly to card write speed.

Chris

[Lpowell]

I doubled the overall bitrate to 86,000,000 and I checked Native 24p.
Okay, I'll try this and see how it affects the recorded bitrates.

[cbrandin]

The recorded bitrates should be about the same because they are governed by Video Bitrate - the only difference is that it doesn't fail with Native 24p checked (I think because it permits a faster instantaneous bitrate) - and the streams look pretty good.

Chris

[NPrincen]

Oh, it all begins to make sense now! If your theory is correct, this explains why AVCHD codec standard is limited to 24 Mbs. If this really requires double the overall bit rate for 24p frames, then this gives a bit rate of 48 Mbs. This is equivalent to 6 MBs, which happens to be the maximum write speed of a class 6 SD card.

So, if you use a class 10 card, then you can safely increase overall bit rate to 80 Mbs. This would allow 24p frames at about 40 Mbs bit rate.

Faster cards like Sandisk Extreme may be able to sustain 20 MBs (short of their advertised 30 MBs). So, maybe these cards can be pushed to 160 Mbs for a 24p bit rate of 80 Mbs.

I may have this all wrong, but the pieces do seem to fit together with your testing and known specs for AVCHD codec.

[cbrandin]

Vitaliy,

I've been looking at the code and the 18000000 constant appears in several places. In about half of them it starts with a value of 9000000, does a compare, and then changes to 18000000 depending on the compare result. The routines look fairly similar. Any idea what these routines are doing?

Chris

[butler360]

Oh, it all begins to make sense now!

The whole GH1(3) project is beginning to resemble this (http://www.coffeecoffeeandmorecoffee.com/archives/filmstills/davincicode.jpg). There's Vitaliy and Cbrandin, chasing down the firmware bugs.

[cbrandin]

Well, it doesn't really need double the bitrate, at least not all the time - the average bitrate shouldn't change much. However, if the camera is buffering entire frames before writing, the Native frames will be twice as big, which could need a faster instantaneous write capability even if the average bitrate is the same. Also, I don't know how smart the flash write routine is; maybe it waits around assuming the card is slow and allows buffer overruns, etc...

One thing I'm fairly certain about - the calculated maximum bitrate limits the speed the camera can theoretically write to flash. It seems to me that the Overall Bitrate should be set to produce a maximum bitrate that matches your memory card.

Chris

[Lpowell]

Sorry, but I saw write-speed limitation failures in FHD mode on a Class 10 Sandisk Extreme 16GB card. I used the 40Mbps High Reliability Patch, modified with Native 24p at 86000000 Overall Bitrate:

http://img413.imageshack.us/i/native24p86moverallbitr.jpghttp://img413.imageshack.us/img413/6335/native24p86moverallbitr.jpg
http://img413.imageshack.us/i/native24p86moverallbitr.jpg/

[cbrandin]

Did you get the failure with Navive 24p checked? I've been testing it all day and I can't get it to fail. Before I could reliably get a failure with Native 24p checked within 10 seconds. What was your test subject? I have to admit I haven't done any testing with Native 24p unchecked so I have no idea about how that might work out. I did, however, test all other modes (SH, H, L, and MJPEG) and got no failures. Stream Parser shows a maximum bitrate of 110,297,900 bps in my tests. Does that match what you got?

I'm not sure the empty frames aren't there for a different reason. They would be hidden with a pulldown. The only way to check is to take a wrapped clip, do a reverse telecine, and step frame by frame to see if there are any where the image doesn't change.

The Overall Bitrate constant exists several places in the firmware, so I suspect a little more research is needed.

I have to admit I'm somewhat confounded - after all, your card is faster than mine.

Chris

[cbrandin]

For example, here are empty frames in a SH mode clip - obviously Native checked has nothing to do with them.

Chris

[cbrandin]

Vitaliy,

I tested a bunch of different modes to try and determine what the 9000000 vs 18000000 constants mean. Nothing changes when I switch between FHD, SH, H, and L modes. This leads me to the conclusion that it might be related progressive vs. interlaced (wrapped) frames, using one value for 60p frames and the other for wrapped 60i frames. Any thoughts?

Chris

[Vitaliy Kiselev]

We need to move lower in AVCHD analizer to understand this empty or almost empty frames.
I can understand why frame can be almost empty after key frame (in static situation).
But why last frame in GOP have some problems is not easy to get.

[Vitaliy Kiselev]

Vitaliy,
I tested a bunch of different modes to try and determine what the 9000000 vs 18000000 constants mean. Nothing changes when I switch between FHD, SH, H, and L modes. This leads me to the conclusion that it might be related progressive vs. interlaced (wrapped) frames, using one value for 60p frames and the other for wrapped 60i frames. Any thoughts?


May be, it is hard to tell.
I am working of some things to allow speed up and improve testing.
It includes IDE modification, special IDC script, plus ability for PTool to use external patches (and use large arrays for them also).
Some file operations will be also opened in next database release.
Plus one safe location (routine that is executed upon Bracketing menu item selection).

[cbrandin]

In the 720p clips the 192 byte frames are devoid of image data. My tests are mostly, but not entirely static. When I look at the clips frame by frame there is absolutely no change - and there should be some changes. I test with a very slow pan on a static scene. I can't explain why they occur after an I frame instead of before, but I assure you they shouldn't be emply in either case. They happen with some regularity in both cases, so I don't think the Native 24p option is causing them - it's more like the wrapped 24p is hiding them.

Chris

[Vitaliy Kiselev]

In the 720p clips the 192 byte frames are devoid of image data. My tests are mostly, but not entirely static. When I look at the clips frame by frame there is absolutely no change - and there should be some changes. I test with a very slow pan on a static scene. I can't explain why they occur after an I frame instead of before, but I assure you they shouldn't be emply in either case. They happen with some regularity in both cases, so I don't think the Native 24p option is causing then - it's more like the wrapped 24p is hiding then.
Chris

In this case we must approach this professionally.
Look only on unmodified 720p footage. And strictly document testing procedures and provide short clips as well as analizing results.
After this we must publish open letter to Panasonic on one of the large techno sites. :-)

[cbrandin]

I'll see if I can make this happen with the factory default firmware, then.

Chris

[NPrincen]

I wonder if you shoot 24p in 60i wrapper and then remove the pulldown, whether you also see empty frames? As you say, the wrapper can be hiding empty frames, but they should reappear if unwrapped. Anybody tried this?

[cbrandin]

I don't know how you could do this. The best I can come up with is to process it with NeoScene and look frame by frame for non-changes. NeoScene will transcode, so looking at the output with an analyzer won't tell. But if you look at the transcoded clip with an editor you should be able to see if nothing changes from one frame to the next.

Chris

[Lpowell]

Did you get the failure with Navive 24p checked? Stream Parser shows a maximum bitrate of 110,297,900 bps in my tests. Does that match what you got?
Yes, my tests were with Native 24p at 86M Overall Bitrate. Stream Parser showed a max bitrate of 110M.

[NPrincen]

Yes, sorry, I forgot that none of the pulldown removal programs put things back in AVCHD, so the analyzer doesn't work. But, I think your suggestion would work. We would need to put it through NeoScene, then look for repeated frames. Unfortunately, I don't have NeoScene, otherwise I would look at this. Thanks for putting up with a suggestion from a noob.

[cbrandin]

Lpowell,

Can you tell me a little bit about your test setup? I would like to try to replicate it as best I can.

Chris

[plasmasmp]

I tried this with
Video Bitrate FHD/SH: 50000000
Overall 86000000

Ran before an after with my old ptools settings which were 62,66,68 and I seem to get the same bitrate shooting the same scene.

Could not get new setting to crash with my patriot lx

[Lpowell]

Can you tell me a little bit about your test setup?
I put the GH1 with a Canon FD 28mm lens on a tripod and point it at dense shrubbery. Then I record 8-sec FHD video clips on the Class 10 Sandisk at 1/60 shutter, f8, and 100-400 ISO. Two out of six attempts failed with a write-speed limitation error, producing a zero-length file.

[donwang]

cbrandin & lpowell;

Just a FYI;

I Tried the following setting :


Get lpowell's latest Tool and 50M setting
http://www.dvxuser.com/V6/showthread.php?t=222791

load setting I then play with the following;
Check 24 fpp native
Overall = 120,000,000 (the Max we can set, it goes with class 10 card)
FHD= 51 then 53 (both looking good, over 53 will turn the rate back to lower rate)
GOP=6 then 4 (default 12 is not as stable as 6, 4 seems more stable than 6)

shot PAPPASART's motion chart more than 60 sec a few times with GOP=4 without problem (produced rate at 46-47)

just let you know put you guys's talent together , it has produced a test result that has never been so good before.

You guys and Vitaliy deserve another round of applaud

Cheers.
don

[divide]

Overall = 120,000,000 (the Max we can set, it goes with class 10 card)


If I'm right, Class 10=10MB/s=80Mb/s, not 120Mb/s.

[alignment1]

This is great news guys--thanks for ongoing developments!!

[svecher]

My tests are mostly, but not entirely static. When I look at the clips frame by frame there is absolutely no change - and there should be some changes. I test with a very slow pan on a static scene.


I put the GH1 with a Canon FD 28mm lens on a tripod and point it at dense shrubbery. Then I record 8-sec FHD video clips on the Class 10 Sandisk at 1/60 shutter, f8, and 100-400 ISO. Two out of six attempts failed with a write-speed limitation error, producing a zero-length file.
Chris, while I haven't tested with new OBR setting yet, it has been my observation from prior testing that "native freeze" happens on very static scenes. I.e. camera locked down on a tripod and pointing at a motionless or slightly moving subject, with majority of scene in focus motionless. Hand-holding or panning the camera, however slowly, let's one dodge the bullet and avoid "native freeze" (in my experience).

[slyn4ice]

Hey guys,

one quick question about testing procedure with Pappas chart of Doom. I have a 24inch monitor and the way i test is I open the chart at 100% and have the monitor fill up the entire frame on the GH1. I have just gone through almost all the suggested settings here and not one gave me a stable 1080p (24p or 60i). Most just crashed after 1-2 sec. So, how is everyone testing? Why is there such a giant difference between cams? I mean not being able to make it crash on Pappas chart? Really? I'm with a Class 10 Sandisk Extreme ... very weird.

[slyn4ice]

Just tested donwang's settings and it crashes immediately no matter what scene I have. 720 just crashes while 1080p freezes the whole cam. Something is fishy here. I suggest if we do test we should give all the information necessary to replicate the experiment - setting file for ptools, camera settings, testing procedure. All of this is pointless otherwise.

[Vitaliy Kiselev]

Guys, I want to warn you that this is thread in development section.
So, if you want to start testing and reporting here, better make new thread in general hack section.

[Oedipax]

Just tested donwang's settings and it crashes immediately no matter what scene I have. 720 just crashes while 1080p freezes the whole cam. Something is fishy here. I suggest if we do test we should give all the information necessary to replicate the experiment - setting file for ptools, camera settings, testing procedure. All of this is pointless otherwise.

+1. It would help a lot if people could give a comprehensive account of their stable 86Mbit settings so the rest of us can start testing it.

[slyn4ice]

Guys, I want to warn you that this is thread in development section.
So, if you want to start testing and reporting here, better make new thread in general hack section.

Got it. I will get a thread going tomorrow in the general section (if nobody hasn't done it yet - wink at OP) as it's night here. No settings will be posted here.

Good night guys.

[cbrandin]

Thanks Lpowell, I'll try some totally static shots with one of my primes. Actually, my test is very similar except that I use a fast shutter (1/500). If there is a breeze outside I just shoot static, if not I pan very, very slowly in order to keep the P frames busy. I've found that shooting with a fast shutter is much harder on the codec than 1/50th.

We may have a ways to go but I think there is something here. When I raised the Overall Bitrate I was unable to get the write speed error with my tests - otherwise I can get failures within seconds, so at least something is getting better. Well, I do know that more of fast cards' bandwidth is being taken advantage of at least.

Chris

[cbrandin]

I know in my enthusiasm I suggested trying other settings. But I agree with others that there needs to be some control here. I want to stick with the Lpowell stable 40Mbs settings for now and drill down the Native 24p issue with just those settings first. His settings have proven (at least, to me) to be utterly reliable and I want to eliminate other stability issues and just concentrate on the Native 24p issue.

Today I plan to first try to replicate Lpowell's failure scenario, and then try to determine if setting the Overall Bitrate higher actually undermines stability with Native 24p unchecked as well.

Chris

[cbrandin]

I tried some totally static, very high detailed scenes and I got a failure with this new patch. Rats - I was hoping this would work better.

There is a definite improvement - but the reliability is not there. It looks like I'll have to do some more digging.

Vitaliy, when the Overall Bitrate patch is set which instance of that constant is changed? Just the one in the initial setup area?

Chris

[Vitaliy Kiselev]

All of them, as far as I remember.

[cbrandin]

All of them, as far as I remember.

Hmmm... now I'm wondering if the routines that start at 9000000 and go to 18000000 based on a conditional need to have the 9000000 changed to 1/2 the Overall Bitrate patch value. What do you think?

Is that what Overall Bitrate 2 does?

Chris

[cbrandin]

I tested the Lpowell settings with the Overall Bitrate raised, but Native left unchecked and I got no crashes. It seems that raising the overall bitrate does not undermine stability by itself.

I also tested the factory default firmware for empty frames in 720p mode. On a static, high detail scene there are quite a few empty frames. It looks like there is some sort of threshold value for changes in the scene that must be exceeded for video data to be produced. So if you have a static scene with something small changing - like a bug moving around - the codec will miss it sometimes. A majority of empty frames (about 60%) occur after I frames. It seems unfortunate to me that there is a threshold as there is little to be gained by throttling what little data would have been produced without it.

When I look for empty frames in 720p mode with the Lpowell settings most of the empty frames occur after every second I frame. This is a strange pattern because if it were simply because of the I frames it seems to me that they would appear after every I frame, not after every second one.

Chris

[c.cook88]

Has this thread moved? I'd like to keep up to date on it.... sounds interesting :)

[R-ik]

It works for me. :cheesy:
stress test in FHD mode with foliage (sorry no grass here)
sd card panasonic class6 4Go and class 4 16Go
pal=>ntsc checked
native 24 checked
video bitrate: 50 000 000
overall bitrate: 86 000 000

MTS files arounds 44000 Kbits/s (this bitrate used to crash before with my class 4 sd card)
framerate 23.976

[DrDave]

I've been digging into the GH1 firmware code and found that overall bitrate appears to have a big effect on the maximum speed that flash memory writes can occur. I’ve always wondered why the write speed error comes up in AVCHD mode when clearly the data rates are well below what the card can handle. I think I found the answer.

When the write speed error comes up it’s because the calculated maximum write rate has been exceeded – whether the card can handle more or not. I tried the Lpowell settings but changed two things: I doubled the overall bitrate to 86,000,000 and I checked Native 24p. Upping the overall bitrate nearly doubled the calculated flash write rate (i.e. the fastest write rate the camera attempts to write to flash). It appears to be working! I’m not getting any lockups and the clips look great. I tested this with highly detailed scenes, fast shutter speeds, everything I normally do to break the codec and it seems to be standing up.

We probably need to do some fine tuning and more testing (hint, hint, Lpowell), but so far it’s looking pretty good. I ran my tests with a Transcend class 10 card.

Chris
Any further progress on this? And why does Native 24p have to be checked, does it affect the data rate as well? So there would be a scenario in which Native 24p would be *more* stable?

[Lpowell]

I tested the Lpowell settings with the Overall Bitrate raised, but Native left unchecked and I got no crashes. It seems that raising the overall bitrate does not undermine stability by itself.

I also tested the factory default firmware for empty frames in 720p mode. On a static, high detail scene there are quite a few empty frames. It looks like there is some sort of threshold value for changes in the scene that must be exceeded for video data to be produced. So if you have a static scene with something small changing - like a bug moving around - the codec will miss it sometimes.
I've seen this behavior in my 50Mbps AVCHD Fast Action Patch as well. While we don't yet have a solution for a reliable Native 24p video mode, your research has enabled me to increase this patch's peak bitrate in FHD and SH videos up to 55Mbps:

http://dvxuser.com/V6/showthread.php?t=222791

I'm also puzzled by the sporatic occurance of empty frames in the middle of an otherwise data-packed 15-frame FHD GOP. It seems extremely unlikely that a single frame in a full-motion video would happen to be virtually identical to the previous frame. Perhaps the GH1 can sometimes fail to complete its sensor scan before encoding of the next frame starts? In that case, the camera would be forced to reuse the data from the previous frame, resulting in an empty P-frame (since nothing has changed). This could be an intentional failsafe mechanism to give the CPU time to catch up when it has fallen behind the framerate.

[cbrandin]

It might be some kind of bitrate failsafe. I don't think so, though, and I hope not. It looks like more of a buffer failsafe. I'm trying to analyze the encoder routines. Hopefully there is something in there that assumes that 24p footage will be wrapped and an output buffer is allocated assuming half-frames instead of full ones. Who knows at this point - but I'm looking!

Chris

[PappasArts]

It might be some kind of bitrate failsafe. I don't think so, though, and I hope not. It looks like more of a buffer failsafe. I'm trying to analyze the encoder routines. Hopefully there is something in there that assumes that 24p footage will be wrapped and an output buffer is allocated assuming half-frames instead of full ones. Who knows at this point - but I'm looking!

Chris

Chris, just curious, with this high bit AVCHD settings you've been working on for 24PN. Is it any more stable using 24 over 60i wrapper rather than selecting 24PN?

If your able to get the bit rate at 80mbits+ for avchd; 24PN or 24PSF, that would be nice if it's stable with either one.

Pappas

[Lpowell]

...Is it any more stable using 24 over 60i wrapper rather than selecting 24PN?

If your able to get the bit rate at 80mbits+ for avchd; 24PN or 24PSF, that would be nice if it's stable with either one.
I did a week's worth of testing on 1080p/60i peak bitrate settings: 55Mbps, 77Mbps, and 110Mbps (as reported by GH13 Stream Parser). The clues I picked up from Chris' research led to a reliable 50Mbps AVCHD patch for Class 6 and higher SD cards:

http://dvxuser.com/V6/showthread.php?t=222791

I can get 77Mbps to produce stable 720p30 AVCHD videos, but not with a 3-frame GOP. At this point, I think 55Mbps is the highest peak bitrate the GH1 can handle at 1080p.

[cbrandin]

The Native mode still has issues - some bugs, I think. Theoretically Native mode should be able to achieve higher bitrates than wrapped 60i mode because it has no duplicate frames created by the telecine conversion that have to be written to flash. There's also the issue of color subsampling - it's a little different in true progressive mode (most like it better) and concerns edges on highly saturated colors. The Native mode is worth fixing because between the higher potential bitrate devoted to unique image data, and the better subsampling, and not having to reverse-telecine clips to edit them; there is quite a bit to be gained.

Alas, you'll have to wait until these bugs can be resolved.

Chris

[cbrandin]

Well folks, I think I have bad news about the Native 24/25p issue. Details here: http://www.dvxuser.com/V6/showpost.php?p=2111347&postcount=57.

Chris

[Roderz]

Well folks, I think I have bad news about the Native 24/25p issue. Details here: http://www.dvxuser.com/V6/showpost.php?p=2111347&postcount=57.

Chris


On the contrary. I think you deserve a big pat on the back for finally solving the problem!

Can I kindly ask: what is your day job and where did you learn the skills to analyze video codecs and the like ?

[cbrandin]

Actually, I might have it wrong. We may not be screwed after all - stay tuned. Many thanks to DaLiv for pointing out my error! I misread an address and went to the wrong places.

As for my day job, I do high tech forensics (processor architecture, encryption, hardware and software analysis, etc...) and testify at trials, etc... A big part of what I do is basically hacking. Also I'm chief scientist for Xpriori (nee NeoCore) - semi retired.

Chris

[tonpole]

As for my day job, I do high tech forensics (processor architecture, encryption, hardware and software analysis, etc...) and testify at trials, etc... A big part of what I do is basically hacking. Also I'm chief scientist for Xpriori (nee NeoCore) - semi retired.

Chris


So your job is finding childporn on peoples computers? I see why you have such a good background on cameras. Is it true that modern sensors include stenographic signatures that allow for a picture to be traced back to a particular camera?

[cbrandin]

No, it's not true. However, it's a common way to hide information within images. When signatures are contained within images and video streams they aren't usually hidden. Well, they are from the image, but the metadata containing that data is typically accessable if you know how to retrieve it. Now you can identify specific sensors sometimes based on dead pixel mapping. Fortunately finding child porn is not part of my job, I'm more in the intellectual property and the anti-trust compliance fields.

Chris

[Vitaliy Kiselev]

As for my day job, I do high tech forensics (processor architecture, encryption, hardware and software analysis, etc...) and testify at trials, etc... A big part of what I do is basically hacking. Also I'm chief scientist for Xpriori (nee NeoCore) - semi retired.


It is very interesting.
You can see more on Chris here - http://www.xpriori.com/management-team
I am impressed.

[Homunculus]

It is very interesting.
You can see more on Chris here - http://www.xpriori.com/management-team
I am impressed.
well shit! with guys like this and vitaly, we're in good hands. we need you two to take a look at gh2 asap and start hacking that puppy! gh13 killed Canons, we want GH2.13 to kill Red! :thumbsup:

[Homunculus]

Actually, I might have it wrong. We may not be screwed after all - stay tuned. Many thanks to DaLiv for pointing out my error! I misread an address and went to the wrong places.

As for my day job, I do high tech forensics (processor architecture, encryption, hardware and software analysis, etc...) and testify at trials, etc... A big part of what I do is basically hacking. Also I'm chief scientist for Xpriori (nee NeoCore) - semi retired.

Chris

keep us updated please! I got my finger on the trigger ready to pull the native 24p OFF if your conclusion is that we can have a STABLE high bitrate codec with 24p native unchecked.

by the way, quick question from a layman. you said possibly that biggest frame size the codec can deal with is 897,152. does this have any correlation or indication of how much resolution gh1 is actually doing? Is it doing true 1080 etc or how many pixels is it really doing or is there no way to know that?

[svecher]

by the way, quick question from a layman. you said possibly that biggest frame size the codec can deal with is 897,152. does this have any correlation or indication of how much resolution gh1 is actually doing? Is it doing true 1080 etc or how many pixels is it really doing or is there no way to know that?
There should not be a correlation. Your question is explored in depth in the following article by Barry Green (http://www.dvxuser.com/articles/article.php/20).

[larryfine451]

So on a personal level Chris ,what is your interest in all this .....Is it the intellectual challenge ?

[cbrandin]

So on a personal level Chris ,what is your interest in all this .....Is it the intellectual challenge ?

Pretty much... Filmmaking and Photography are hobbies of mine. A long time ago I studied photography (the technical aspects of it) at RIT.

Chris

[KurtF]

Bump.

Any further news on 24pn?

[Vitaliy Kiselev]

Chris and I had been very busy last days.
But we work on reversing.

[MarekV]

Nice! Hopin' for next amazing achievements around GH1 and 24pN guys! And take your time of course ... :)

[Ian-T]

It is very interesting.
You can see more on Chris here - http://www.xpriori.com/management-team
I am impressed.Geez....all of a sudden I feel very stooooopid!!! :) Impressive indeed.

[HHL]

Heh...that's FREAKING AWESOME Chris. Congrats on being a genius. :) I love this forum man!

[Willyfan]

As seems is impossible (actually) solve the 24p issue for buffer size problem, I suggest to wrap in a different wrapper: not 60i but some similar to a 48i.
In PAL, I can record at 50i instead 25p, and I have the same data to encode. Yes, there is some little difference in interlaced vs progressive encoding, but not so big at this point.
So, if we can record a 48i file, may be we have no more the buffer problem,and data to encode is the same than 24p but lesst than 60i with better quality.
If this is possible, this is not a solution but a workaround for reliablity. Or not?

[zcream]

Pah! Harry Direen's moustache is far more impressive than Chris.

Heh...that's FREAKING AWESOME Chris. Congrats on being a genius. :) I love this forum man!

[omnidecay]

As seems is impossible (actually) solve the 24p issue for buffer size problem, I suggest to wrap in a different wrapper: not 60i but some similar to a 48i.
In PAL, I can record at 50i instead 25p, and I have the same data to encode. Yes, there is some little difference in interlaced vs progressive encoding, but not so big at this point.
So, if we can record a 48i file, may be we have no more the buffer problem,and data to encode is the same than 24p but lesst than 60i with better quality.
If this is possible, this is not a solution but a workaround for reliablity. Or not?

I've actually thought something like this also...However, I am not sure how hard it would be to create a type of frame rate in camera then program it to pulldown automatically.

[aquavideo]

well shit! with guys like this and vitaly, we're in good hands. we need you two to take a look at gh2 asap and start hacking that puppy! gh13 killed Canons, we want GH2.13 to kill Red! :thumbsup:

Isn't this site owned by Jarred? I like a lot of things about the GHs and particularly the hacks, but killing RED isn't one of them!

[omnidecay]

I might have stumbled upon something and thought it might be useful...I was thinking about modular's settings/cbrandin's double theory and it got me to playing around. With these settings I was able to record the papa chart-o-death and will supply the .mts files if needed. Both times I was able to record the chart for a minute with 24p checked.

FHD- 32xxxxx
Overall- 90xxxx
GOP- unchecked

FDH- 40xxxxx
overall- 80xxxx
GOP- 15

I tried dropping the GOP down to 10 but I received a card write error in 5 sec. I am using a sandisk 8gig class 10 30 MB/s...My thinking here (and not on a deep level) is if we shoot in FHD but keep the bits down but keep the overall bitrate very high. This might produce some higher quality images....Both were averaging around 25-35MB/s a second and I was using the media information on VLC to show me consistently updating stats.

On a side note, when I used the star chart I was recording it off of my 28" 1920x1200 monitor so that might have effected the outcome as well. I dont have access to a high rez printer to get it off the computer so it was the best I could do. With these findings it makes me think the AVCHD codec has two points |-----------| and it will bounce its bitrate between those until it cant take it any more and write error but between those bookends is infinite. The MJPEG seems to work in steps where it will shoot for the high step and then go down to the next step if it cant handle it.

http://vimeo.com/16495828

For anyone who wants one of the files...this was filmed with: overall- 90xxx, FHD- 32xxx

Shutter: 1/60
ISO: 100
F-stop: 2

nostalgia: sat: -0, contrast: -2, sharpness: -2, red: 0

[pradip850]

Hi guys I am new to dvxuser, I have just brought GH1 yesterday with 1.2 firmware. I did hear about the hack with ptool and I tried doing it myself but it seems like the hack is not working? (Or I am not doing it right!!). Though it's my first time using DSLR I am a quick learner, I did search on the net for step by step instructions on how to hack GH1 (I did lots of forum reading too but still was unsucessful)....

I hope by posting my question, it will help me solve my problems as well as many others that may or are facing similar issues.....So can somone please be kind enough to help me/us out by posting clear instructions here (I guess everyone once went through this, so you know what it's like guys, so help would be very much appreciated)..

Ok i will explain what I've done so far, I went to the following link for instructions http://wiki.eoshd.com/index.php/Begi...Firmware_Guide (http://wiki.eoshd.com/index.php/Beginners_GH1_Custom_Firmware_Guide) I've followed and done exact steps that were mentioned in this page and few additionals. Additionals were ticking the "Thrid-party battery" box, ticking the "30min limit removal" box and ticking the "Video Bitrate H" and changing the value to 50,000,000 from 16,000,000 under it (Even though ticking and changing the value for "Video Bitrate H" wasn't mentioned in that page, I done it simply because of the image shown there had a tick in it and some forum reading done before)......By the way lots of forum i read talks about using ptool v3.35 and v3.36 but the ptool I used was v3.40d...not sure if that makes any differences?

Then I saved the firmware naming it GH1_133.BIN (yes with double underscore). Then I copied the saved GH1_133.BIN to empty Class 10 Transcend 16GB memory card and inserted into my GH1. After that I clicked "Playback button" in GH1 and it automatically asked me if I wanted to update to version 1.3 which I selected "Yes" then it started to load and after loading, GH1 restarted (Now at this point, I assume that the hack has been already applied).....Piece of cake!! (that's what I thought!! But how wrong was I?)

I then started to record some videos (without changing any setting in the menu, maybe i have should have? If yes, please let us know what settings to change).....Then I took the memory card out of my GH1 and inserted into my card reader and plugged into USB port and extracted the recorded videos (Can I extract the videos from my GH1 straight using the usb cable that came with GH1? instead of taking the memory card out? If yes, how? Because I tried doing it, but my PC didn't find any USB connection, maybe because I should have installed the software that came with my GH1? (which I didn't) I simply didn't do it because I thought it was going to update the firmware version automatically and I dind't wanted to take the risk of having the latest firmware updated in my GH1 where I wouldn't be able to apply the hack if I am right?? and I did make sure that I only buy GH1 with specific firmware so I can apply the hack).....

After looking at the video shot from my GH1 I was sad :(....(because it had very similar quality comparing to my £200 camcorder where as I spent £800 for GH1).....I also compard the videos which I shot earlier without trying to apply hack and the quality were very similar??.....So guys this is my sad story (LOL)....please help me/us out :(

Thank you very much for taking your time to read this. Help and answers to all questions asked above would be very very very much appreciated.....

Now I helplessly wait for a super man :P...hahaha good day :D

[omnidecay]

Testing some more I was able to get stable settings using my sandisk extreme class 10 30MB/s WITH 24p checked:

Overall bitrate: 90xxx
FHD: 40xxx
GOP: 15

I was able to film papas chart here: http://vimeo.com/16509731

[butler360]

Testing some more I was able to get stable settings using my sandisk extreme class 10 30MB/s WITH 24p checked:

Overall bitrate: 90xxx
FHD: 40xxx
GOP: 15

I was able to film papas chart here: http://vimeo.com/16509731

I really hope you're right, but the camera never seemed to be fully still. A lot of the problems came from a complex scene, a dead still camera, and a wide lens.

[omnidecay]

I was using 50mm 1.4 canon FD...I did the best I could to rest the camera on the back of a chair. I think I did a total of 30 tests with different settings all using the GOP of 15. I failed left and right except for certain variations of the settings I posted. I also moved the camera around/changed focus a bit since movement also caused some of the tests to fail. I posted so someone who is much more advanced then I am can examine my findings. I know the goal is to keep GOP numbers low but is it possible that higher GOP settings improve 24pn stability? The quality was still pretty good and will be able to post more test shots over the weekend.

[butler360]

Try putting it on a solid tripod so there's absolutely no movement.

[omnidecay]

I dont have time tonight but I will sometime soon. What is the advantage of locking off vs moving the camera? I would think movement would tax the camera more then just focusing onto a still frame. If its better to keep the camera still. Why did papa add the ability for the picture to spin?

[butler360]

I guess the theory is that movement can blur the image a bit. So a super-detailed still image is clearer and thus results in more detail. I guess the spinning part was for testing not only still images, since that was the big problem at the time (and still is for 24P) and what everyone was focusing on.

[humpman]

Testing some more I was able to get stable settings using my sandisk extreme class 10 30MB/s WITH 24p checked:

Overall bitrate: 90xxx
FHD: 40xxx
GOP: 15

I was able to film papas chart here: http://vimeo.com/16509731
i tested this settings with papas chart, and when camera on hands it works nice, but when i start to shoot on tripod, camera shows "1 sec" and thats it

[omnidecay]

i tested this settings with papas chart, and when camera on hands it works nice, but when i start to shoot on tripod, camera shows "1 sec" and thats it

OK so the tripod does make a big difference...I was at a club/film premiere all weekend and was able to shoot 24p no problems handheld with the above settings. This issue seems to be completely backwards and illogical but I am probably just thinking about it the wrong way. If you take a computer graphics card for example. It will be taxed way more if images are moving vs standing still because the GFX has to create every single frame. Where as with a camera its more taxed not moving...I understand motion blurred images are easier to resolve due to less information to create the frame. But what I don't understand is why an image would throw the codec completely whack once it has been able to capture for a few seconds or longer.

It seems like the codec is continuing to try and resolve the image and with each resolve throw more bit rates at the image (like a never ending loop) causing an eventual write error. I do not understand or possess the technology to view AVCHD footage after the camera has crashed and deleted the footage. It would be very interesting to see how the bitrates/frames react while filming a complex non-moving image over time until crash. Maybe a safety buffer has to be coded in like a stop loss?

[omnidecay]

i tested this settings with papas chart, and when camera on hands it works nice, but when i start to shoot on tripod, camera shows "1 sec" and thats it

Actually, now that I think about it. You might need to cover the lens before filming an crazy scene like the chart. What happens is the camera throws a ton of bitrates at the image right away and it overloads the sensor. Try placing your hand in front of the camera when you hit record and then wait a second or two and slowly pull your hand away. I wish there was a way to tell the camera to not freak out like that and have it slowly increase bitrates when needed vs dumping them all at once. In the first second of filming I will get the highest bitrate and then it will level out. IMO this seems like a very weird way to have a camera handle its image processing.

[butler360]

Yeah, this hand trick has been discussed before, not sure in which topic, though.

[Lpowell]

What happens is the camera throws a ton of bitrates at the image right away and it overloads the sensor... I wish there was a way to tell the camera to not freak out like that and have it slowly increase bitrates when needed vs dumping them all at once. In the first second of filming I will get the highest bitrate and then it will level out.
Initial bitrate is exactly what "Preset Bitrate" and "Preset Bitrate 2" allow you to control. In my patches, I set both of these values to 180 to throttle back the data size of the first I-frame in the recording. That guards against almost all cases of recording failures immediately after pressing the movie button.

[noirist]

Initial bitrate is exactly what "Preset Bitrate" and "Preset Bitrate 2" allow you to control. In my patches, I set both of these values to 180 to throttle back the data size of the first I-frame in the recording. That guards against almost all cases of recording failures immediately after pressing the movie button.
What are the highest quality Native 24p settings that you would recommend?

[Lpowell]

What are the highest quality Native 24p settings that you would recommend?
Sorry, but I can't recommend any Native 24p settings. Nothing that I've tried has produced reliable results.

EDIT: Until now! Check out my latest patch:

Blackout-Powell Native 24p Patch - GH1 FTW!!! (http://www.dvxuser.com/V6/showthread.php?243813-Blackout-Powell-Native-24p-Patch-GH1-FTW%21%21%21)