Compared: F3, Alexa, Red, 35mm, 16mm and 1D.

For the 3rd time on this thread I gotta agree with Giuseppe. Did you learn that you need to underexpose the F3 by a stop? I don't think that's true at all. You just need to expose it properly. I don't underexpose my F3 at all. Truth is we know nothing about what led to the F3 being apparently overexposed. We don't know the gamma, we don't know how they set exposure - whether they used a light meter or a monitor, etc. If they used a light meter to expose video that's a huge mistake from the get go.

I do know that I will probably have to field doubts from someone who saw this and thinks the F3 looks lousy compared to a Canon, and that the problem is the 8bit codec - all of which is false.

If a comparison test isn't done very carefully I don't think it tells you very much and just tends to create false impressions which unfortunately can give a camera a stigma. I'm also very dubious about the Zacuto tests because they appeared to be done very carefully but left out all the tests where the DSLR's would fall apart - probably part of the "care" I'm afraid.

That said - it ius very hard to do them well and you can learn something from them if you have a very skeptical eye. But it is dangerous to give them too much weight.
 
I think many of us in this room have the background to mentally adjust for variances in camera testing such as the overexposure of the PMW-F3 in this test, to where we can easily agree that there's even more performance that can be milked out of it when properly calibrated.

But, not everyone who looks at camera tests is a DP or engineer who can so easily intuit these quirks. Indeed, Camera Operators, Directors, Producers, and students of the craft all look at these sort of tests, and they hope to glean from it what they should be shooting their projects on (if they have their druthers, of course). We in this room understand the fallacy of making decisions soley on the basis of such testing, particularly when, as Lenilenapi points out - some testers may leave out tests that would clearly demonstrate the limitations of certain cameras.

But others won't get this. They'll watch a test where a PMW-F3 in REC709 with factory defaults is put against a Canon 5D MkII - by an operator familiar with the 5D, and using the F3 for the very first time - and they'll walk away wondering why someone would ever pay $14k for such a camera.

Camera tests are inherently flawed, but this only makes it that much more important that tests are done carefully, variables are documented, and each camera is presented in it's best light. Soon we'll have the 4:4:4 & Slog option for the F3. I would suggest that future camera tests inclusive of the F3 ought to use two F3 cameras - one in Slog, and one without - and both tweaked at production and post for best possible results. And of course, you could add a third distinction - internal 4:2:0 versus external recording. Or 4:4:4 versus 4:2:2. So many options... no wonder anyone who isn't a DP gets confused...
 
Watching tests is one thing. Performing your own is another. I suggest everyone do their own testing.
 
TimurCivan said:
Watching tests is one thing. Performing your own is another. I suggest everyone do their own testing.
Click to expand...

+1 as this is what I do. You learn SO much when comparing a camera you know EXTREMELY well to a new camera you actually want to make work the way you work. In that regard, I've enjoyed it. I am doing that now with my FS100 and enjoying it.
 
I have a better idea...stop shooting tests and shoot some GIGS! 99% of all our clients would be happy for a shoot with any of them. I never had a client ask for a test-shoot or a chart yet. ;-)
 
TimurCivan said:
Watching tests is one thing. Performing your own is another. I suggest everyone do their own testing.
Click to expand...

Absolutely everyone needs to test their camera, find the right settings, etc before doing real work. (I cringe when I see posts like "I get my new camera Thursday and I have a live event on Saturday, what do I do?")

But I think when you take this test, plus Philip Bloom's real world test (FS100, F3, AF100, 5D), plus some other tests together you start to get better ideas of what things are consistently happening. (AF100 has more of a tendency to clip highlights for instance. For the F3 rather than saying underexpose its better to say 'use zebras and histograms, not light meters, to expose.)

Then you have a better idea what things you especially need to test when you get your camera.

Besides, maybe you all have 35mm, an Alexa and a Red to play with. Most people don't. If I didn't see something like this I'd never see how they compare. I was pleased with the F3 performance and the fact that it can do better. I've made the mental note to double check exposure.

As I said, I really want to see how F3 S-log compares to Alexa Log-C. And I won't have an Alexa available for that either. I hope someone does it.
 
Last edited:
LeavingTheCandy said:
I have a better idea...stop shooting tests and shoot some GIGS! 99% of all our clients would be happy for a shoot with any of them. I never had a client ask for a test-shoot or a chart yet. ;-)
Click to expand...
ha ha hallelujah brother! I was beginning to think these cameras were made for the sole purpose of doing tests... If you don't own an Alexa then it doesn't really matter how it compares. Bottom line with the F3 is that if you shoot it right, the image is going to exceed expectations, which means you will have exceeded expectations and at the end of the day everybody is happy. "Get a job Lebowski!"
 
TimurCivan said:
Watching tests is one thing. Performing your own is another. I suggest everyone do their own testing.
Click to expand...

Problem is not a lot of people have the time or money to get an Alexa, Red, F3, F35, Film, and DSLR all together at one time. So its not easy to do so. My only questions are... If you do have the time and money to get all of those cameras together for a shoot to compare why wouldn't you spend the extra time to correctly setup, expose, document, and grade the footage? Kind of a waste in the first place to do so dont you think?

I'm not trying to bash anyone I'm simply sticking to facts. Fact is this "test, shootout, comparison" whatever you wanna call it, it doesn't help the viewers learn anything at all. Its only the DP's comments on here crying out 'hey I know something was wrong' that is giving the useful information. This video without our comments attached to it gives false visual representation of each camera. There are standards, and their are ways to do tests. I've done my own testing and published them online, but I try to give real world information, even down to the way I compressed it out of FCP, what settings, what bit rate, and usually provide the super massive "RAW" files from the camera or at least a super high bit rate ProRes file for people to download so they aren't seeing issues of vimeo compression/colorshift/gamma changes.

Its the only way to actually provide useful information to others. I may seem like the whistle blower at times, but I'd rather be called that then just let the information out there not be represented right. I have a hugely deep passion for cinematography as a whole, and I'm seen trends right now that are hurting our young/new comers... They arent learning right from wrong, they are just learning brand names and simplicity. Knowing your painbrush(camera) is so important. But false info about that paintbrush leads to less people using the right brush for the job... That then trickles down to less quality product to show, which if you REALLY wanna get crazy about it... ends up lessening our craft over time, lighting doesn't become as important to the viewers because they get dumbed down by the quality they see on youtube and vimeo... Then producers dont want to spend as much or hire the right people... productions as a whole become a lesser quality and thus our high standards drop and craft suffers.

I know this is a very extreme way of looking at it, sounds crazy even. But there is truth to it, I see it as we speak. I see producers insisting on hiring a DP that "owns a DSLR" instead of insiting on hiring a "good DP". The camera has nothing to do with it. The DP DOES. I'd trust my F3 in the hands of Roger Dekins way before some DSLR "master" student.
 
The problem I have is people take these tests as fact in so many locations. I had a disagreement with a local filmmaker about the read/reset speed of the AF100. He was quoting Alan Roberts paper and I put my name behind the statement that Alan Roberts made a mistake on his skew measurements. I took his images that he provided to others and redid the calculations and got results much more correct than what his calculations came out with. I even tried to talk with Alan Roberts about the mistake, but of course, he will not talk to people anymore since most people attack him about ANYTHING he says. (And on one hand, I don't blame him, but on the other hand, his information is incorrect). Not only did I redo my calculations multiple times, I even had another person use a different method altogether to also generate read/reset times so that we could see what that method produced, and once again, I got the same values as I had calculated, which was a method developed by Adam Wilt who I've spoken with frequently on it. Then the person I was having a disagreement with had mentioned that an article on ProCoalitionVideo also said the read/reset was slower just like the Alan Roberts report. So, I wrote the author of the article and lo and behold, they just quoted the white paper by Alan Roberts and didn't bother to do their own test. And this was something that isn't subjective, rolling shutter is easy to measure. Imagine how these image comparisons get propagated to others.

Why do I do these tests? It's not for the numbers, it's for the people who are coming from other cameras and want to understand where limitations are before they buy, just as many of these tests are trying to do (and I can understand that). I discovered the rolling shutter on the AF100 was EXTREMELY close to shooting the RedOne at full resolution. I found that it is twice as fast as the 5D. This is important. Watching someone swing a camera back and forth and seeing skew in the image is what everyone else was doing, saying things like "Looks the same as the GH2 to me" which it was NOT (unless you are in the crop mode). In fact, what I discovered is the actual jello effect (expansion/compression in the vertical domain is very hard to get on the AF100 compared to all the DSLR cameras). Skew still exists when panning, but people don't think about it. If you pan at a give speed and the pixel difference between the top and bottom is 100 pixels on a vertical line, if you increased the read/reset to twice the speed, the line is still going to be tilted by 50 pixels. Because of this, and the fact people do not pan consistent speed like they think, they don't realize this.

I tried to take the subjective out of the equation so someone suffering from rolling shutter would know that the AF100 would be an improvement possibly from what they are coming from.

Tests like this that are subjective are of course even harder, but the reason people get on here and pick them apart is because people WILL quote this stuff as the ABSOLUTE TRUTH. I see it all the time. If nothing else, it's an educational experience as well. Sometimes a tester finds out they were doing something wrong that others spot. Sometimes they discover things that people didn't know. If another user hadn't done an extensive AF100/7D test I wouldn't have caught on so quickly about the exposure differences between them because I just hadn't thought about doing that test.

I know some people feel like they are being attacked when they post up tests, and at times, I will ABSOLUTELY agree that some people go beyond polite discourse (and we try and catch that), but at the same time, when I post something up, I expect it to be scrutinized because *I* want to be accurate in my statements as well. That's one of the great things about DVXUser is that we spent time vetting out information from others and ourselves.
 
Did anyone else notice the light roll off on the lamp at 16:00-16:28? Both the 16mm and 35mm film cameras looked smooth. The only camera that came close to matching the film cameras (to my eye) was the Canon 1D. All the rest of the cameras had a harsh yellow band in the lamp. Why did the Canon do so well (in this one regard) compared to the other digital cameras? Was it because of its sensor size?
 
Shootouts are extremely popular. And they're also nearly impossible to pull off. I've organized several. I've tried to do it the way Giuseppe says -- I got six cameras together, with six camera experts (one per camera). Got fricassee'd and filleted for it. I've tried doing comparison tests between three cameras, just me and Jarred working them. Got accused and barbecued. I've tried doing tests myself so there are no variables. Got called every name in the book.

Simple fact is: it's very, very, very difficult to do a worthwhile comparison PROPERLY. You'd have to be exceptionally skillful and spend a lot of time learning both units, and doing a comparison with three doesn't make it a little harder, it raises the complexity by an order of magnitude. And another order of magnitude to compare four. And there are no rewards for doing it -- you will be lambasted and called every name in the book, by whoever's "chosen" camera "loses".

I have always published detailed methodology on every test I've ever conducted, so that anyone who has questions would be free to recreate it and see exactly how I got my results, and if I did something wrong, they could point out the error in the process. Nobody ever does; they just complain.

It is a thankless task and one I hope never to do again.

Yet it's all anyone wants to see. They get so excited with any sort of "versus" thread, or "comparison", or, especially, "shootout".

There's only one comparison that I know of that did it completely right and was absolutely and completely unimpeachable, and that was the SCCE. And part of the reason they could do it right is that they had 140 people working on it. They had ASC cinematographers supervising, they had expert operators assigned to every camera, they used the same lens on every camera for the entire shoot, and they were absolutely meticulous and diligent. It was easily the best comparo I've ever seen, and while there were plenty of ways it could have been done wrong, they had so many top-notch top-flight folks on it, there was so much redundancy and so many skilled eyeballs and veterans working it, they caught just about every mistake and did everything right.

If you want to see truly how the F3, Alexa, Red MX, Canon DSLRs, and 35mm film compare, I highly recommend getting to a screening of the SCCE.
 
Barry_Green said:
If you want to see truly how the F3, Alexa, Red MX, Canon DSLRs, and 35mm film compare, I highly recommend getting to a screening of the SCCE.
Click to expand...


I was going to bring up that test, problem is no one can get their hands on one, it was never made public. I've seen the short skinny of the test but not the whole thing (which was available to ASC members - and I believe still is)

That is the test above all tests and that is the one I wish people could see, to show how these things should be done. If someone wants to do a camera test, why not combine all the dudes who wanna do camera shoot outs, specially the ones who are paying for it, and they ALL get together and just lump it into one definitive test... Come to think of it... I could organize something of a larger scale with sponsors and the whole 9... But would anyone want to put themselves into the firing range as all the other testers have?

I'm game for organizing one just as big as out in LA with SCCE, but we'd have to get a lot of great op's and DP's to join in.
 
The SCCE is planning on making a blu-ray of the test results, and they organize screenings as per the website.

The only way the SCCE happened was 100+ people volunteered, and they got Zacuto to pay for ut (while simultaneously giving Zacuto zero editorial influence; the deal was, as I understand, Zacuto received the right to make a documentary on the test and that was all). And, as Bob Primes has pointed out, the results are already out of date... the F3 now has S-Log, the FS100 is now available, etc.
 
"And, as Bob Primes has pointed out, the results are already out of date"
Click to expand...
That's hitting the nail on the head.
Look at what has been released the last 12 month's.
Its impossible to have a shootout that's really up to date.
Its back to the old "Do It Yourself"
With your own touch and feel. Some people like their shots a bit bright, others underexpose and lift in post.
With digital the margins are very very thin. Impossible to really convey that in a test that has to be understood by everybody in the business.
 
Bump

Bump

Wendt said:
Did anyone else notice the light roll off on the lamp at 16:00-16:28? Both the 16mm and 35mm film cameras looked smooth. The only camera that came close to matching the film cameras (to my eye) was the Canon 1D. All the rest of the cameras had a harsh yellow band in the lamp. Why did the Canon do so well (in this one regard) compared to the other digital cameras? Was it because of its sensor size?
Click to expand...


Anyone?
 
BannedinDV said:
But would you trust yourself with the F3 over Roger Deakins with a DSLR? ;)
Click to expand...

hahaha I bow down to Roger, I get to talk to him a couple of times a month to pick his brain. He could blow anyone out of the water with a DSLR, though I know Shane Hurlbut, ASC is probably more suited to kick ass with a DSLR, he I believe is the only ASC member actually trying to push DSLR shooting as much as he is. Tho the yoda-ness of Roger might still win over. haha
 
You must log in or register to reply here.
Back
Top