Data Lossless Compression

[visionmind]

Hey,

I'm really excited about the possibility of onboard 4k recording at 27MB, but I'm wondering if there are any plans for a mid-point data rate, like a data lossless compression, inbetween the 27 and the 323MB data rates?

There are some unknowns of course, so this is only a question. I don't know if it would require too much processing power and or other technicalities that I'm unaware of.

The idea/suggestion/question would really be applicable to next-gen hardware (like eSata) with higher datarates (without RAID), because the current data rate (27MB) pretty closely maxs out a 3.5 external harddrive (according to Jared Lands article)

Thanks,

Visionmind

[David Newman]

Lossless compression will involve compressing noise, which Red shows not a huge amount of, but it is still there, introducing an issue for lossless compressor. For real world image data (studio shots with black or green-screen background will be easier) you can expect 10-bit lossless wavelet compression to be about 3:1 (at best, worse for 12-bit linear.) Yet 110+ MB/s is quite doable on a 4 drive RAID. The question might be why you feel that lossless is needed, as 27MB/s wavelet compressed RAW will look very nice.

[Graeme_Nattress]

David is right. We did some tests for lossless compression and got around 2.5:1 quite reliably, however, for the size of file, just over 1MB / frame, the RAW looks bloody amazing, and really is the thing to use for practically all practical purposes.

Lossless is lovely, but really, is it necessary. What is necessary is that the amount of compression you use is not visible as artifacts even through the post production process and out to distribution.

Graeme

[visionmind]

Yeah, it's not that lossless is necessary (for most work), more than anything I was just wondering if an inbetween was being planned (27MB and 323MB), not necessarily specifically lossless (although that was the example I gave).

Also I have yet to see the moving footage (although the stills do look nice) so I'm not aware if an inbetween could even be created (if there is little to no difference between the uncompressed and the redcode RAW).

No matter what you guys have done some amazing stuff and I trust you'll continue doing so.

Thanks for your replies,

visionmind

[Brook Willard]

I think we're all just fearing that 27MB/s is just too good to be true... that the image will come out muddy and noisy. Now obviously you guys would never herald REDCODE as the standard unless it really was groovy enough to cut it...

It's like having a DSLR that either shoots low-quality JPG or RAW... everybody wants something in between. I'm not trying to equate REDCODE to a low-quality JPG... but the difference between data rates is almost dramatic enough to make the uninitiated feel frightened.

Or something.

If REDCODE isn't good enough but RAW is overkill... what will we be left with?

Show me some REDCODE, baby!

[Graeme_Nattress]

I'd be lying if I said that REDCODE RAW is as good as uncompressed RAW. However, REDCODE RAW looks bloody amazing and I'd defy anyone to spot the compression in it until you do diabolical things to the image, like really set the levels so far out of whack it destroys the image.

Graeme

[donatello]

" just fearing that 27MB/s is just too good to be true"

4k+ camera $17.5K - too good to be true ?
18-85 cine zoom 9.5K - too good to be true ?
300mm 2.8 lens 5k - too good to be true ?
4k at 27MB/s - too good to be true? ..

do NOT doubt the RED team !! if it isn't good they will not release it !!
4k Redcode Raw will be too good and it will be true !

[Kjetil Haugen]

Hi Graeme.

We have faith in you!

Just wondering if this new footage you guys are shooting these days are shot using the current Redcode raw or uncompressed... Will the next images/videos posted in the gallery be meant to show off the sensor or the code? Or both?

[zakforrest]

http://www.dvxuser.com/V6/showpost.php?p=684668&postcount=4

[Gibby]

4k+ camera $17.5K - too good to be true ?
18-85 cine zoom 9.5K - too good to be true ?
300mm 2.8 lens 5k - too good to be true ?
4k at 27MB/s - too good to be true? ..

do NOT doubt the RED team !! if it isn't good they will not release it !!
4k Redcode Raw will be too good and it will be true !

Donatello...layin' down the prose!

Well stated...

Gibby
RED #8
www.cut4.tv
www.4umat.com

[Graeme_Nattress]

I'd be lying if I said that REDCODE RAW is as good as uncompressed RAW. However, REDCODE RAW looks bloody amazing and I'd defy anyone to spot the compression in it until you do diabolical things to the image, like really set the levels so far out of whack it destroys the image.

Graeme

[Graeme_Nattress]

We're working on moving to a REDCODE RAW workflow internally.

Graeme

[gunleik]

Well, that's sorta promising -;)

I've had some fun (and unexpected corrupted shots) with the FS-100 on the HVX the last few days.

To have a company taking care of the whole workflow, codec, recording gear etc - except your choice of NLE - is kinda reasuring.

Guess this might come from my Mac heritage, but nevertheless.

Gunleik

[Brook Willard]

do NOT doubt the RED team !! if it isn't good they will not release it !!
4k Redcode Raw will be too good and it will be true !
Just for the record, in the same post you quoted me from:

Now obviously you guys would never herald REDCODE as the standard unless it really was groovy enough to cut it...

[dchenevert]

David is right. We did some tests for lossless compression and got around 2.5:1 quite reliably, however, for the size of file, just over 1MB / frame, the RAW looks bloody amazing, and really is the thing to use for practically all practical purposes.

Lossless is lovely, but really, is it necessary. What is necessary is that the amount of compression you use is not visible as artifacts even through the post production process and out to distribution.

Graeme well if could have only one (uncompressed RAW, or REDCODE RAW),
I would take REDCODE RAW.

If you are going to offer 2 4k formats, I would choose REDCODE RAW
and ... lossless-compressed RAW. I.e., if you are going to offer
uncompressed RAW at all, why not lossless-compress it first?

and of course my preference would be all 3 (uncompressed RAW, lossless
compressed RAW, REDCODE RAW).

regarding "why offer lossless or uncompressed RAW?"...
the early use of this camera will probably include a certain amount of
"gee where does this little artifacty thing come from?". If you cannot
see the original truly-raw bits, then "caused by REDCODE compression"
will always be a possible (even if unlikely) culprit.

[stokestack]

If you are going to offer 2 4k formats, I would choose REDCODE RAW and ... lossless-compressed RAW. I.e., if you are going to offer uncompressed RAW at all, why not lossless-compress it first?

I think this is a good question. 2.5:1 sounds pretty good to me, when you're dealing with such a data payload. Doubling your storage capacity and cutting the strain on the pipes seems like a no-lose proposal. I guess if there's enough processing power to do wavelet compression on the fly, there's enough to do lossless, right?

[David Newman]

stokestack,

Yes and No. Wavelet encoding to lossless (using CineForm wavelets as an example) is the same compute load for lossless as lossy for the transform part only. The problem is you have lot of bits to entropy encode, and you can't quantize that data to a more manageable size (for lossless.) So the compute load climbs with bit-rate. Hardware implementations have the same issue but with a hard limit, hardware entropy encoders choke at the high data rates. So you so have to over-engineer if you want lossless, to handle the extreme bit-rate cases of 2:1, this means more CPU or higher clock rates on your entropy coders. If you get visually lossless on half the CPU power == longer battery life, or simpler hardware, it might not be worth the over-engineering.