Pixel Shifting the F35 - DVX100 Andromeda & HVX200 Style :)

jenningsp

jenningsp

Well-known member
So I had a thought this morning...

Does anyone remember the DVX Andromeda and the never released HVX Hydra system from RealStream?

They basically took the uncompressed RGB stream from the SD sensor and did a proper Pixel Shift process (similar to what the HVX does internally) to get a true HD luminance channel.

Because it had individual sensors for each of the red green and blue channels, they used the 444 information and sub pixel misalignment to work it out. And they managed to get real results.. Going from 773×495 to 1546×990 on the DVX and 960x540 to 2100x1090 on the HVX

That's basically a 4x increase in resolution on paper. If only it were a 2x increase in reality. It did work.

The F35 has individual red green blue pixels does it not? Does the uncompressed RGB image out of the HDSDI reflect the sensor data? IE keep the slight pixel misalignment information???

So basically I am thinking we could potentialy pixel shift our way from a 1080p image to a true 2160p image :)

Here is a modern photoshop tutorial with the same basic principals - http://petapixel.com/2015/02/21/a-practical-guide-to-creating-superresolution-photos-with-photoshop/

What do you guys think? Is it possible to make a after effects script that does this? does a plugin already exist? is it worth it?

Fun! :)
 
I've essentially said this is possible (to an extent) for a long time. Because the F35 is a RGB stripe sensor, the relative positions of the Red, green, and blue channels are knowns, so when you record an RGB color space stream you're essentially recording three mapped photosite positions per RGB pixel. Mind you, that's only half of what the sensor sees (six mapped positions per RGB pixel output), but it's still enough to do complicated math and get a resolution gain. But that gain comes at a loss of color fidelity. You're trading known "real" data for interpolated guesses. In some cases the results might be worth it, but in others, not so much.

Yes, an algorithm could be written. But I'm not sure how far it would really get you.
 
You guys are starting to hurt my brain. I have no idea why you would ever want to upres F35 footage, unless for a final 4K/UHD deliverable. In which case a good Lanczos filter or something will probably do just as good a job.

The F35's sensor was built to do one thing and one thing only - to provide the finest 1080P 4:4:4 footage possible. And they did that.

The DVX is a totally different layout. If you wanted to go down this path, I'd start with an F23 at least!
 
It doesn't help that Sony have themselves abandoned their principles and started labelling things like the F5 "4K", although I cannot blame them - they were fighting a losing battle. You can't cure stupid.
 
oh i'm completely with you and think uprezzing is a waste of time for everyone involved. This more a theoretical topic and not actually about uprezzing... more figuring out if there is some hidden/forgotten potential for real actual resolution increases... ya dig?
 
I dig. Of course your first argument is going to be defining "real actual resolution increases", in which case there is not much to gain.

The F23 doesn't even have pixel-shifted sensors, which was a trick used by things like the Z1 etc... to increase apparent resolution. Because the pro cameras do not use that trick to begin with, I don't think there is much to gain.

The nearest you would get is in some broadcast cameras (eg. F900 etc) there was only ever 4:2:2 available out of the camera, but 4:4:4 was available internally - Sony eventually took advantage and built the studio cameras to use that feed. The F23 did much the same.
 
You should also know that the F35 uses a unique vertically striped sensor design that utilizes six individual photo sites to create one resulting pixel. To attempt to create an algorithm that reworks the mapping of these photo sites would be both a massive undertaking and one that is likely to have a lot of inherent issues. And in the end it would only apply to this small handful of cameras.

You might get an increase in pixel count resolution, but then you would need to fudge the numbers away to deal with all of the physical offsets creating massive moire issues. Potential is three times the data load for perhaps a 20% increase in resolving power, if you were really lucky.

So no, not worth it in many ways.
 
Mitch Gross said:
You should also know that the F35 uses a unique vertically striped sensor design that utilizes six individual photo sites to create one resulting pixel. To attempt to create an algorithm that reworks the mapping of these photo sites would be both a massive undertaking and one that is likely to have a lot of inherent issues. And in the end it would only apply to this small handful of cameras.

You might get an increase in pixel count resolution, but then you would need to fudge the numbers away to deal with all of the physical offsets creating massive moire issues. Potential is three times the data load for perhaps a 20% increase in resolving power, if you were really lucky.

So no, not worth it in many ways.
Click to expand...

Agreed. I was aware of this, but didn't always explain it fully. The RGB stream gives you mapped placements for sets of two of those six pixels, but they aren't square photosites. That means horizontially you have 5760 mapped luma signals to interpolate from, but only 1080 vertically. One method of upscaling would be to scale to 5760 by 3240, but this isn't a standard delivery size. So you'd either have to crop or downscale for delivery, and that kills part of the "benefit" off the bat. A second method would be to scale to 3840x2160 by using each original RGB set to create four new RGB sets by clever interpolation with adjacent locations. Even if you did that, you'd still run into complaints from those who see real 4k as being 4096 - which there is no practical way to scale to from the F35 sensor without lossy cropping.
 
If they had arranged the sensor a different way you would have been able to possibly do the trick Thomson did on the Viper and "group" pixels, but Sony/Panavision decided against that. There will be a reason why although I don't know it (probably in the interest of keeping photosites a relatively useful shape)
 
I would have thought that the easiest way to knock this on the head would be the fact that all the processing has been done by the time the signal leaves the sdi port - so you couldn't do anything there, you'd have to get inside the camera and put your algorythms in the processing pipeline somewhere no? Good luck with that! :)
Steve
 
steve phillipps said:
I would have thought that the easiest way to knock this on the head would be the fact that all the processing has been done by the time the signal leaves the sdi port - so you couldn't do anything there, you'd have to get inside the camera and put your algorythms in the processing pipeline somewhere no? Good luck with that! :)
Steve
Click to expand...

Not exactly the case here, although it would apply for most cameras. With bayer sensor cameras the video signal is the interpolation of multiple photosites, so reversing back to get photosite data isn't possible.

But the F35 is an RGB stripe sensor. Which means it has vertical rows of each color, always repeating in sequence. And each output RGB pixel comes from 2 red, 2 green, and 2 blue photosites. If you record an RGB signal (not YUV) you're recording a separate red,green, and blue value for each pixel location in the 1920x1080 frame. For display it's interpreted as if these all are overlaid on one another, one location. But you can really map them - let's say, hypotetically it's Red, Green,Blue from left to right on the sensor. Essentially, in an RGB signal you have 5760x1080 mapped luma values - but with one third color filtered for red, one third color filtered for blue, and one third color filtered for green. That's essentially a 6k output in grayscale - but it's not square pixels.

So we have the mapped locations. We don't have to get inside the camera. We just have to perform calculations to correct for the color filters by sensibly detecting edges and making best guesses about intermediate pixels.

So, you can do it, but the point remains - there's not enough advantage to make it worth the bother.
 
Or, just use the F35 for what it is. :)

The label doesn't really matter. HD, not HD, 4k, 4K. The camera resolves upwards to 2.5K and produces the best color in the business.

The F65 is newer, is 4K, but you would have to hold me at gun point to have me saying I prefer the F65 over the F35 or the way it renders faces, highlights, color and motion over the way the F35 does. What matters is the overall feeling of the image. Numbers are cheap these days.

If the F35 would start spitting out anything other than a HD file by means of some hack it wouldn't really change what the camera is. It would be again just for bragging rights. If it would mean a compromise in color to obtain "extra lines of resolution" to fit a container specification, no thanks. I will always choose color over resolution.

I have said this before but I prefer a camera which promises HD and actually delivers more than that than a camera that promises 4K but never delivers it, while compromising in every other front, such is the case with every 4k camera out there besides the F65.

The F35 is fine the way it is.
 
Aaveum said:
Or, just use the F35 for what it is. :)

The label doesn't really matter. HD, not HD, 4k, 4K. The camera resolves upwards to 2.5K and produces the best color in the business.

The F65 is newer, is 4K, but you would have to hold me at gun point to have me saying I prefer the F65 over the F35 or the way it renders faces, highlights, color and motion over the way the F35 does. What matters is the overall feeling of the image. Numbers are cheap these days.

If the F35 would start spitting out anything other than a HD file by means of some hack it wouldn't really change what the camera is. It would be again just for bragging rights. If it would mean a compromise in color to obtain "extra lines of resolution" to fit a container specification, no thanks. I will always choose color over resolution.

I have said this before but I prefer a camera which promises HD and actually delivers more than that than a camera that promises 4K but never delivers it, while compromising in every other front, such is the case with every 4k camera out there besides the F65.

The F35 is fine the way it is.
Click to expand...

word!
 
You must log in or register to reply here.
Back
Top