Results 1 to 10 of 10
  1. Collapse Details
    Standards & Guidelines for testing
    #1
    Junior Member
    Join Date
    Apr 2010
    Posts
    8
    Default
    Hi guys,

    the testing work / footage here still seems a bit random. What kind of scenes, lighting, objects do you want to see in comparison ? What list of settings would be worth testing?

    I would like to help testing, but I need directions. What I can offer is good light (TV Studio), a days work and maybe a model :-)


    Reply With Quote
     

  2. Collapse Details
    #2
    Senior Member
    Join Date
    Jun 2009
    Posts
    677
    Default
    It depends what kind of testing you are wanting to do.

    Some testers are looking for reliability - how high can the numbers be set (etc) before problems arise, eg camera locking up? In this testing there's a degree of assumption that higher settings are (a) better and (b) better enough to be worth using.

    Some testers are looking at what quality improvements, compared to lower settings or to original firmware, can be obtained from given sets of Ptool settings. Here I perceive (rightly or wrongly!) that the testing isn't as scientifically valid as one might like. For instance, you would have to take footage of the same scene under the same circumstances (including lighting conditions) and then carefully compare before and after. Until quite recently this wasn't very easy, but now that you can prepare firmware-only cards with different setups, and use these to set the camera up for testing each setting in the field in the course of about five minutes, then you've got a better chance of filming a level playing field (figuratively or literally!).

    Then there's issues relating to panning or to movement within a static shot. The movement has to be repeatable. The panning has to be at the same speed each test. Not that easy to do. I've suggested various ideas before - and not yet tried them myself really, I confess - eg shooting something rotating on a turntable at a consistent speed, or maybe even putting the GH1 on a steadily rotating turntable so it pans at a steady rate. Or shoot across a road with fine foliage on the opposite side, and cars passing at a probably-steady speed.

    I'm not sure how much has really been determined in the way of analysis of diminishing returns - in other words, upping the settings by 50% might make the picture 50% better, but does upping them another 50% make the picture another 50% better, or only another 10% better, though using up the card another 50% quicker. (Obviously you can't actually give percentage values, I'm just illustrating a general point).

    Now due to the huge number of posts in this firmware hack forum, I may have missed some examples which would provide methodical analysis, in which case perhaps someone might put together some links to the best properly conducted tests and I'll eat my words (not for the first time...)

    Maybe the best way to test is to imagine that you are trying to prove to a biased 5DII owner that the GH1 hack makes it the superior camera...

    Good luck with the testing!


    Reply With Quote
     

  3. Collapse Details
    #3
    Senior Member
    Join Date
    Aug 2006
    Location
    Colorado Springs
    Posts
    473
    Default
    Controlled tests are good but they only test the codec under controlled circumstances. I run a lot of these and they are valuable, but it's not enough.

    My best stress test - the one that can most reliably "break" the CODEC - seems almost silly. I have some flowers (real little ones) underneath some rather ratty bushes in my back yard. I have my wife stand off to the side slowly waving one hand up and down in the edge of the frame. This test will break the CODEC faster than any controlled test I've been able to contrive so far.

    My 2 cents worth is to suggest doing controlled tests mixed with random shots. At the very least, the random shots will occasionally manifest something the controlled shots don't, making you re-think your controlled tests.

    Chris
    Last edited by cbrandin; 06-23-2010 at 07:33 AM.


    Reply With Quote
     

  4. Collapse Details
    #4
    Senior Member
    Join Date
    Aug 2006
    Location
    Colorado Springs
    Posts
    473
    Default
    One thing I've discovered that really stresses the CODEC is to shoot highly detailed subjects at a fast shutter speed. The fast shutter retains detail because there isn't any motion blur. A detailed scene shot at 1/50 might work fine where if you shoot the same scene at 1/500 the camera gets a write error, for example.

    Chris


    Reply With Quote
     

  5. Collapse Details
    #5
    Default
    Chris is spot-on with that one, high shutter speed + extreme detail = broken codec. Which, frankly, isn't that big of a worry to me, as long as it doesn't break at 1/60th, but yes that's definitely a solid observation.

    Controlled/repeatable results are key towards determining what's happening and whether it's getting better or worse.

    I believe right now Vitaliy is asking people to concentrate on the sensor mode settings, and figuring out what they do. If you can set up a repeatably-shootable scenario (a dedicated still life with controllable lighting perhaps?) and then explore and report on the sensor mode settings, that would probably be quite helpful.


    Reply With Quote
     

  6. Collapse Details
    #6
    Senior Member plasmasmp's Avatar
    Join Date
    Dec 2008
    Posts
    508
    Default
    The strange thing is that even a broken setting might not be broken. Everytime I've had a write error, I try to duplicate the problem. Around 20% of the time I'll have another error, but any subsequent times of trying the same shot, or the other 80% of the time I will be able to pick up and keep recording the same shot after a power down.

    There are so many variables that were going to need more in depth testing methods to find out whats going on.

    Also, I'm with Barry. For testing the settings I wont go higher than 1/60 because there really is no purpose for me to ever shoot at a higher shutter speed.


    Reply With Quote
     

  7. Collapse Details
    #7
    Senior Member
    Join Date
    Aug 2006
    Location
    Colorado Springs
    Posts
    473
    Default
    All the shutter speed does in these tests is preserve detail. If you have a truly static frame (almost) 1/50th will have the same effect as long as some things are moving slightly - it's just harder to set up a test to do that. I would not assume that just because you don't shoot at fast shutter speeds that testing that way isn't a good way to manifest conditions that will be present even at 1/50th under certain conditions. There isn't really a systematic way to test for success because you are trying to prove a negative - that there are no errors. You can only systematically test for failure, and make adjustments according to your knowledge about where and when failures happen.

    Chris
    Last edited by cbrandin; 06-23-2010 at 11:33 AM.


    Reply With Quote
     

  8. Collapse Details
    #8
    Default
    Quote Originally Posted by plasmasmp View Post
    The strange thing is that even a broken setting might not be broken. Everytime I've had a write error, I try to duplicate the problem. Around 20% of the time I'll have another error, but any subsequent times of trying the same shot, or the other 80% of the time I will be able to pick up and keep recording the same shot after a power down.

    There are so many variables that were going to need more in depth testing methods to find out whats going on.

    Also, I'm with Barry. For testing the settings I wont go higher than 1/60 because there really is no purpose for me to ever shoot at a higher shutter speed.
    Really? Isn't the standard shutter angle 180 degrees and when shooting 720P wouldn't that be approximately 1/125 second? I suppose not going beyond 1/60 makes sense if your frame rate never goes beyond 30 fps. But clearly many people shoot at 60 fps.

    Further, the 180 degree shutter is simply a guide. There may be good reasons to shoot faster - though I tend not to like the results of doing so.


    Reply With Quote
     

  9. Collapse Details
    #9
    Senior Member
    Join Date
    Aug 2006
    Location
    Colorado Springs
    Posts
    473
    Default
    I just posted a tiny tutorial on testing electronic things (something I have a lot of experience with) that might be of interest: http://www.dvxuser.com/V6/showthread...09#post2028309

    Chris


    Reply With Quote
     

  10. Collapse Details
    #10
    Senior Member plasmasmp's Avatar
    Join Date
    Dec 2008
    Posts
    508
    Default
    Quote Originally Posted by wturber View Post
    Really? Isn't the standard shutter angle 180 degrees and when shooting 720P wouldn't that be approximately 1/125 second? .
    Yes. I should have clarified by saying 180 degree shutter. I meant for 24p alone. For 60p I use 125th. For 24p I alternate between 1/50 and 1/60 depending on the light source.


    Reply With Quote
     

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •