Standards & Guidelines for testing

B

bauhausler

New member
Hi guys,

the testing work / footage here still seems a bit random. What kind of scenes, lighting, objects do you want to see in comparison ? What list of settings would be worth testing?

I would like to help testing, but I need directions. What I can offer is good light (TV Studio), a days work and maybe a model :-)
 
It depends what kind of testing you are wanting to do.

Some testers are looking for reliability - how high can the numbers be set (etc) before problems arise, eg camera locking up? In this testing there's a degree of assumption that higher settings are (a) better and (b) better enough to be worth using.

Some testers are looking at what quality improvements, compared to lower settings or to original firmware, can be obtained from given sets of Ptool settings. Here I perceive (rightly or wrongly!) that the testing isn't as scientifically valid as one might like. For instance, you would have to take footage of the same scene under the same circumstances (including lighting conditions) and then carefully compare before and after. Until quite recently this wasn't very easy, but now that you can prepare firmware-only cards with different setups, and use these to set the camera up for testing each setting in the field in the course of about five minutes, then you've got a better chance of filming a level playing field (figuratively or literally!).

Then there's issues relating to panning or to movement within a static shot. The movement has to be repeatable. The panning has to be at the same speed each test. Not that easy to do. I've suggested various ideas before - and not yet tried them myself really, I confess - eg shooting something rotating on a turntable at a consistent speed, or maybe even putting the GH1 on a steadily rotating turntable so it pans at a steady rate. Or shoot across a road with fine foliage on the opposite side, and cars passing at a probably-steady speed.

I'm not sure how much has really been determined in the way of analysis of diminishing returns - in other words, upping the settings by 50% might make the picture 50% better, but does upping them another 50% make the picture another 50% better, or only another 10% better, though using up the card another 50% quicker. (Obviously you can't actually give percentage values, I'm just illustrating a general point).

Now due to the huge number of posts in this firmware hack forum, I may have missed some examples which would provide methodical analysis, in which case perhaps someone might put together some links to the best properly conducted tests and I'll eat my words (not for the first time...)

Maybe the best way to test is to imagine that you are trying to prove to a biased 5DII owner that the GH1 hack makes it the superior camera...

Good luck with the testing!
 
Controlled tests are good but they only test the codec under controlled circumstances. I run a lot of these and they are valuable, but it's not enough.

My best stress test - the one that can most reliably "break" the CODEC - seems almost silly. I have some flowers (real little ones) underneath some rather ratty bushes in my back yard. I have my wife stand off to the side slowly waving one hand up and down in the edge of the frame. This test will break the CODEC faster than any controlled test I've been able to contrive so far.

My 2 cents worth is to suggest doing controlled tests mixed with random shots. At the very least, the random shots will occasionally manifest something the controlled shots don't, making you re-think your controlled tests.

Chris
 
Last edited:
One thing I've discovered that really stresses the CODEC is to shoot highly detailed subjects at a fast shutter speed. The fast shutter retains detail because there isn't any motion blur. A detailed scene shot at 1/50 might work fine where if you shoot the same scene at 1/500 the camera gets a write error, for example.

Chris
 
Chris is spot-on with that one, high shutter speed + extreme detail = broken codec. Which, frankly, isn't that big of a worry to me, as long as it doesn't break at 1/60th, but yes that's definitely a solid observation.

Controlled/repeatable results are key towards determining what's happening and whether it's getting better or worse.

I believe right now Vitaliy is asking people to concentrate on the sensor mode settings, and figuring out what they do. If you can set up a repeatably-shootable scenario (a dedicated still life with controllable lighting perhaps?) and then explore and report on the sensor mode settings, that would probably be quite helpful.
 
The strange thing is that even a broken setting might not be broken. Everytime I've had a write error, I try to duplicate the problem. Around 20% of the time I'll have another error, but any subsequent times of trying the same shot, or the other 80% of the time I will be able to pick up and keep recording the same shot after a power down.

There are so many variables that were going to need more in depth testing methods to find out whats going on.

Also, I'm with Barry. For testing the settings I wont go higher than 1/60 because there really is no purpose for me to ever shoot at a higher shutter speed.
 
All the shutter speed does in these tests is preserve detail. If you have a truly static frame (almost) 1/50th will have the same effect as long as some things are moving slightly - it's just harder to set up a test to do that. I would not assume that just because you don't shoot at fast shutter speeds that testing that way isn't a good way to manifest conditions that will be present even at 1/50th under certain conditions. There isn't really a systematic way to test for success because you are trying to prove a negative - that there are no errors. You can only systematically test for failure, and make adjustments according to your knowledge about where and when failures happen.

Chris
 
Last edited:
plasmasmp said:
The strange thing is that even a broken setting might not be broken. Everytime I've had a write error, I try to duplicate the problem. Around 20% of the time I'll have another error, but any subsequent times of trying the same shot, or the other 80% of the time I will be able to pick up and keep recording the same shot after a power down.

There are so many variables that were going to need more in depth testing methods to find out whats going on.

Also, I'm with Barry. For testing the settings I wont go higher than 1/60 because there really is no purpose for me to ever shoot at a higher shutter speed.
Click to expand...

Really? Isn't the standard shutter angle 180 degrees and when shooting 720P wouldn't that be approximately 1/125 second? I suppose not going beyond 1/60 makes sense if your frame rate never goes beyond 30 fps. But clearly many people shoot at 60 fps.

Further, the 180 degree shutter is simply a guide. There may be good reasons to shoot faster - though I tend not to like the results of doing so.
 
wturber said:
Really? Isn't the standard shutter angle 180 degrees and when shooting 720P wouldn't that be approximately 1/125 second? .
Click to expand...

Yes. I should have clarified by saying 180 degree shutter. I meant for 24p alone. For 60p I use 125th. For 24p I alternate between 1/50 and 1/60 depending on the light source.
 
You must log in or register to reply here.
Back
Top