AVCHD -- Native vs. transcoding

[rpcx]

As an HMC150 newbie, I have been confused by the discussions about native AVCHD support vs transcoding. I understand the performance issue for computers which lack the horsepower for native AVCHD editing. But what about the downside of transcoding? The usual complaint seems to be the nuisance of an extra step in one's workflow. But nobody seems to mention loss of quality from the additional compression-decompression cycle.

All the practical video compression algorithms are lossy. That means image quality is degraded with each cycle. The high cost of video storage forces us to accept two compression cycles -- one for capture and (if we edit the source material) an additional one for delivery. Adding an intermediate codec for editing sounds like a terrible idea. Yes, you might do it as a temporary expedient to solve a CPU performance issue. But you should warn people that it is a bad idea and tell them to switch to native just as soon as the continuing march of Moore's Law overcomes the performance issue. Right???? That makes Apple's family of ProRes codecs an elegant implementation of a bad idea???

I suspect that I am missing something here. There ought to be a good agrument for transcoding in general and Apple ProRes in particular. Would someone please provide it.

 

[ATL Media Group]

Valid points. What I (and many others) also appreciate about editing in CS4 is the seamless integration between the products in the collection. I can edit right off of the SDHC card (and play the timeline back to external monitor with no additional hardware) and if I want to do some composting in After Effects.. I use dynamic link to open the Premiere sequence in After Effects. Do my composting/CC/Title... whatever then go right back to premiere... When it's done.. I dynamic link again and open the sequence right in encore. I have yet to render a file BTW. Author my Blu-Ray and build disc.. finally rendering in Encore. I can also render to DVD or flash right from Encore.
It's EXTREMELY useful and efficient. Rendering ONE time is wonderful!

 

[Mike Harvey]

...That makes Apple's family of ProRes codecs an elegant implementation of a bad idea???

I suspect that I am missing something here. There ought to be a good agrument for transcoding in general and Apple ProRes in particular. Would someone please provide it.

You make it sound like back in the days of editing SVHS or Hi-8 deck to deck. It's not. Transcoding in and of itself isn't a bad idea, it's what you transcode to and how you do it. Depending on how you do it, there will be minimal, if any loss of quality.

Most people here seem to be very happy with the ProRes implementation Apple uses for FCP, and it seams to do a really good job. As a Vista x64/CS4 person, I haven't had really any complaints about using the DVCProHD transcoder Panasonic gives you. It could be better, but it's not bad. I've also tried transcoding to the free AVID DNxHD codec and thought it did a decent job. However, I'm realistically going to invest in Cineform sometime soon. What little I've tinkered with the Neo Scene and Prospect trials have left me impressed. The only complaints I've heard have had to do the bugs associated with CS4. The codec itself is spectacular, and ultimately probably gives better results than editing AVCHD natively.

Plus, editing native doesn't mean you're going to end up being lossless... especially with a long-GOP codec. If at anytime you need to render out another file to be edited later, you're probably going to take a small quality hit... even if you export to AVCHD/MP4/M2T, since you're now going to have new I-frames and such (someone please correct me if I'm wrong about it... I'm not an engineer).

As to your other points... it just depends on every individual's workflow. Taking the time to transcode isn't a big deal for me... I just set it up right before I go to bed, and do it while I sleep (or go to work, or what ever).

As to the "High cost of video storage"... I can pick up a terabyte harddrive for under $100. That's pretty darn cheap if you ask me. I remember buying a 300 GB drive for $300 only a few years ago.

 

[Barry_Green]

If at anytime you need to render out another file to be edited later, you're probably going to take a small quality hit... even if you export to AVCHD/MP4/M2T

Why would you render intermediates to AVC? If it's something you're going to be rendering as an intermediate, you'd be better off rendering to CanopusHQ or DNxHD or something like that.

it just depends on every individual's workflow. Taking the time to transcode isn't a big deal for me... I just set it up right before I go to bed, and do it while I sleep (or go to work, or what ever).

Right, but as you said, that doesn't work for everyone -- for someone who has to deliver a same-day-edit or is working against a deadline, it can be a huge detriment to their workflow.

I think the quality of the transcode isn't normally that much of an issue; modern editing codecs like prores or dnxhd or canopushq or cineform are all really pretty good. Voltaic's recompression to mpeg4 is not what i consider satisfactory, but the quality level of the others is fine. The space issue is much more of an issue; yes storage is getting cheaper but shooting 21mbps footage and seeing it expand up to 12x or larger just to edit it, blowing up to 220mbps or 270mbps or whatever... seems absurd.

I think rpcx's point bears examination:

That makes Apple's family of ProRes codecs an elegant implementation of a bad idea???

This battle has been fought before, over and over, and the result is always the same: new format comes out, it's "too hard to edit", it gets transcoded, and then as soon as the editing stations catch up, they abandon transcoding and support the native format.

I started many years ago, back when Matrox was offering its DigiSuite lineup, you'd convert everything to its matrox I-frame MPEG hardware. You couldn't even play a stream of DV without hardware acceleration. But it wasn't long before every editor was native DV.

Then we had the same thing with HDV -- it's too hard to edit, so up comes Lumiere and Cineform and other intermediate codecs. But where are we now? computers are fast enough, so everyone edits native hdv.

So here we are again, the same thing is being said: "it's too hard to edit, we have to transcode, transcoding is wisdom". I say give it a year, and when CUDA and ATI Stream are properly supported and delivering hardware acceleration of the decode/encode process, that'll be the end of transcoding to an editing codec.

But does that make Apple's implementation a "bad idea"? Well, there's the question. I think it's a temporary band-aid and no more. But there's a good case to be made that Apple editors NEED that band-aid right now. It's a process that solves a problem. And while I expect that graphics card acceleration will be more ubiquitous a year from now, what are they supposed to do between now and then?

I personally don't think there's much future in this transcoding approach, it's been done repeatedly and subsequently abandoned every time it's been tried, and I think history will repeat itself again. But it's a valid and viable solution for today, it solves a short-term problem, and therefore I don't know that you could fairly call it a "bad idea" so much as an "elegant implementation of a temporary inconvenience". Presuming, of course, that Apple intends to take advantage of hardware acceleration as Adobe is and as Mark Von Lanken is intimating that EDIUS may be doing shortly.

 

[Mike Harvey]

Why would you render intermediates to AVC? If it's something you're going to be rendering as an intermediate, you'd be better off rendering to CanopusHQ or DNxHD or something like that.

I was refering editing AVCHD native, and for whatever reason having to export a file that would itself also be edited later. Sorry if that wasn't clear.

 

[Superfresa66]

That Mr. Green would of been a great post a few days ago in the other thread... Having a band-aid to help the painful bleeding experience of AVCHD native editing! =)

 

[Barry_Green]

I was refering editing AVCHD native, and for whatever reason having to export a file that would itself also be edited later. Sorry if that wasn't clear.

No, I got that -- but that's still not how I'd work. I mean, is there any reason you'd want your intermediates to be AVC? Every editor I use lets you freely mix and match pretty much any format, any frame rate, any codec, all simultaneously on the timeline. So I'd edit native AVC, but if I had to render a section out I'd probably want to choose the highest-quality intermediate codec for that section...

 

[sewolla]

As to the "High cost of video storage"... I can pick up a terabyte harddrive for under $100. That's pretty darn cheap if you ask me. I remember buying a 300 GB drive for $300 only a few years ago.[/quote

But you need to pick up two of them, one for project files one for backup. At about a gig a minute after transcoding, that 1TB drive gets me only 14 + projects. Plus you need a second 1TB drive for every one you buy for back-up. Therefore a 1 hour shoot requires about 120GB's of space. So if you stay fairly busy and your projects take about an hours worth of filming (an average shoot for me) you will soon be spending a LOT on hard drives that really start to clutter up your workspace.
If editing AVCHD files natively saves me a significant amount of HD rel estate, (not sure how big a diff there is in file size) that's reason enough to modify my workflow to edit natively.

 

[pailes]

There is one aspect everyone seems to ignore:
If you apply just one single filter to your footage (may it be color correction or a simple sharpen/blur filter) the NLE needs to re-encode anyway when it goes out of the NLE. And when that happens you're better off with a codec which is optimized for multigeneration encoding and long gop AVC is not such a codec.

Who of you is only cutting stuff together?

 

[ATL Media Group]

There is one aspect everyone seems to ignore:
If you apply just one single filter to your footage (may it be color correction or a simple sharpen/blur filter) the NLE needs to re-encode anyway when it goes out of the NLE. And when that happens you're better off with a codec which is optimized for multigeneration encoding and long gop AVC is not such a codec.

Who of you is only cutting stuff together?

Okay... I can plop down a Magic Bullet filter with CC, put in a transition or two... and play it back in RT at reduced quality *for editing*.. When I'm ready to go to Blu-Ray.. just link to Encore and make the disc. No transcoding, no rendering until I go to disc.

 

[PerroneFord]

But you need to pick up two of them, one for project files one for backup...

Ok.

At about a gig a minute after transcoding, that 1TB drive gets me only 14 + projects....

WTH? Why are you storing 14 projects an a single hard drive? THAT'S your issue right there. Do a project, archive it, start another. You shouldn't have anything on that hard drive that are not actively editing.

So if you stay fairly busy and your projects take about an hours worth of filming (an average shoot for me) you will soon be spending a LOT on hard drives that really start to clutter up your workspace.

So archive off to LTO or BluRay like I do. Fairly cheap and easy.

If editing AVCHD files natively saves me a significant amount of HD rel estate, (not sure how big a diff there is in file size) that's reason enough to modify my workflow to edit natively.

Well if that works for you, then great. But you also need to understand that a lot of folks in this industry don't originate on handycams, and codecs like ProRes, DNxHD, etc. offer MASSIVE reductions in file size over native editing. You are coming to this game where you are capturing in a delivery format, and editing in a pro editing format. Very different workflow.

If you and your clients don't care about the quality lost in editing an 8-bit lossy codec, then why worry? Cut native and deliver.

 

[PerroneFord]

I suspect that I am missing something here. There ought to be a good agrument for transcoding in general and Apple ProRes in particular. Would someone please provide it.

Uncompressed 8-bit HD is about 430GB per hour if I remember correctly. Your HMC150 is recording what... 20GB per hour? THAT is where the real loss is. Nearly 95% of the signal. These professional intermediate codecs you are calling "lossy" have so much more bandwidth than your recording format, calling them lossy is silly. You could put your footage through 10 generations of ProRes HQ, render to BluRay, and A/B that with your native footage out of the camera on the same TV, and from 5ft never tell the difference.

As far as transcoding in general, you are talking about a workflow developed for people who's capture format EXCEEDED the intermediate codec. So the intermediate codec offered reduced space requirements and faster workflow. Our handycams capture FAR less information but we use the same workflow because of computer speed issues.

ProRes transcoded footage has won how many Oscars now? Been nominated for a couple dozen? Something tells me, that your technical needs won't be beyond those of the Coen Brothers for some time. Use it with confidence.

 

[j1clark@ucsd.edu]

The space issue is much more of an issue; yes storage is getting cheaper but shooting 21mbps footage and seeing it expand up to 12x or larger just to edit it, blowing up to 220mbps or 270mbps or whatever... seems absurd.

The Wife is hard of hearing, and we got permission to capture a Flash class she took at a local community college on a camera. We bought a cheep Sony HDRSR11 from Costco.

Well, other than in camera playback to our TV, a 32 in. Flat panel, when I 'transcoded' the AVCHD to AIC, I ended up at the end of the semester with almost 2T of AIC media... The original AVCHD for the semester fit on to just a little over the 60G internal drive of the camera, and so I did have to backup that to archive, but it was a simple file transfer...


As it was, we found it was just easier to go through the material playing with the cameras replay capability... I had thought I'd 'edit' the material to cut out gaps in presentation, or the like to tighten it up...

For my narrative film of last year, most of the 'shots' were a minute or less, so the transcode time for that was small... but still the resulting AIC files where huge... One strange thing on that, I shot using 24p mode... when my FCE transcoded to AIC, it retained the 24 fps, but of course FCE doesn't have a 24 fps setup capability... so I used a 25 PAL setup...

 

[j1clark@ucsd.edu]

As an HMC150 newbie, I have been confused by the discussions about native AVCHD support vs transcoding. I understand the performance issue for computers which lack the horsepower for native AVCHD editing. But what about the downside of transcoding? The usual complaint seems to be the nuisance of an extra step in one's workflow. But nobody seems to mention loss of quality from the additional compression-decompression cycle.

In 'native' editing there is no encoding-recoding pass. The NLE records frame addresses, if you will, were significant editing operations are to be performed, then when the editing is finished, and the results exported, the output encoding is performed on the 'original' media. Of course in that moment there is a 'decode-encode' but it is at most 1 generation.

While I've not done this with DV or other moving picture codecs, I did a quick test on recoding JPEG... you know, the 'awful' still image encoder that is 'so lossy', that all 'real' photographers ran, not walked, to 'raw' mode...

Well, I found that for my test cases, after 7 or 8 Jpeg recompressions, perhaps at that point 'loss' would be visibly apparent to the casual observer... otherwise, 1, 2, or 3, Jpeg recompressions added only a few 'bits' of noise to the image, well under the 'human observable' threshold...

Of course where Jpeg does fall off, and why someone would want 'raw' is because of the initial compression and lost data... but once the image has been mapped to the 'compressed' form, subsequent compressions don't 'lose' as much as that initial compression.

 

[pailes]

Okay... I can plop down a Magic Bullet filter with CC, put in a transition or two... and play it back in RT at reduced quality *for editing*.. When I'm ready to go to Blu-Ray.. just link to Encore and make the disc. No transcoding, no rendering until I go to disc.

I'm often swapping footage between different programs to achieve something. Sometimes I do just some color correction in my favourite color correction tool and then I move the clip to some other program for compositing etc. How am I supposed to do that with AVCHD? Even if all the tools of my choice would support AVCHD, I would simply not want to render my stuff to AVCHD and move that stream to a different program. For such things I'd like to have a more robust codec.

 

[ATL Media Group]

I'm often swapping footage between different programs to achieve something. Sometimes I do just some color correction in my favourite color correction tool and then I move the clip to some other program for compositing etc. How am I supposed to do that with AVCHD? Even if all the tools of my choice would support AVCHD, I would simply not want to render my stuff to AVCHD and move that stream to a different program. For such things I'd like to have a more robust codec.

That's my point. In the past I'd render to an editing friendly robust codec (CanopusHQ) then import the footage into each other app. With CS4 I don't need to render! Period! The best part is if I want to make a change in ANY of the apps I don't need to re-render! Just make the change and viola.. it's changed in the other apps on the fly!

Beautiful!

 

[j1clark@ucsd.edu]

I'm often swapping footage between different programs to achieve something. Sometimes I do just some color correction in my favourite color correction tool and then I move the clip to some other program for compositing etc. How am I supposed to do that with AVCHD? Even if all the tools of my choice would support AVCHD, I would simply not want to render my stuff to AVCHD and move that stream to a different program. For such things I'd like to have a more robust codec.

One of the 'models' of processing shot film to screened movie, is part of the work flow involves digital intermediaries. I think this model is appropriate for some set of workflows involving narrative film making.

I think the 'drop in and use' workflow is better suited for certain types of ENG, perhaps Event coverage, especially if there's a need for 'just after the event' delivery, etc.

In the case of a narrative film, there are perhaps months (nay... years...) of postproduction work to bring the film to the screen. So, in that case, saving a 'few minutes (or hours)' of transfer to a 'digitial intermediary' is not a significant cost.

In the case of the 'storage explosion' required, well, that can be 'managed' in that one can buy enough storage to handle one's 'immediate work'... it may be unfortunate to have stuff 'offline' and have to 'put it up', as one use to have to do with all computer systesm... (Multics had the concept of a tape warehouse and a 'robot' to retrieve tapes, and automatically load them as required... but I digress...)...

So, for someone doing heavy manipulation or editing, I don't see the 'native' mode as being so great a benefit as to cause one to lose sight of the goal... which is produce a film...

On the other hand for an ENG operation... having the media 'ready to go to broadcast switches'... well, then 'native' does have it's benefits... but then one may not be doing heavy duty manipulation... and so a splice and dice capability may just be sufficient.

Since I've for the most part done mostly After Effects, for my 'fancy' stuff... I'm pretty use to limited RT playback, like setting my work area right around an effect or transition, and long render times to see the resulting fully rendered sequence (and iterations thereupon...).

 

[pailes]

That's my point. In the past I'd render to an editing friendly robust codec (CanopusHQ) then import the footage into each other app. With CS4 I don't need to render! Period! The best part is if I want to make a change in ANY of the apps I don't need to re-render! Just make the change and viola.. it's changed in the other apps on the fly!

I'm talking about different tools from different vendors. I really don't want to limit myself to Adobe CS4 or whatever.

 

[mcsmooth]

That's my point. In the past I'd render to an editing friendly robust codec (CanopusHQ) then import the footage into each other app. With CS4 I don't need to render! Period! The best part is if I want to make a change in ANY of the apps I don't need to re-render! Just make the change and viola.. it's changed in the other apps on the fly!

It really is sweet, I have to say that anyone with a 150 not using this NLE is really missing out! Transcoding isn't the worst thing in the world... but after switching to 4.1, I would NEVER go back! Editing is just much more fun again.

I'm not saying everyone needs to switch their NLE (Premiere has it's share of annoyances), but at the current time, Adobe has one of the best no-nonsense workflows for AVCHD.

I'm talking about different tools from different vendors. I really don't want to limit myself to Adobe CS4 or whatever.

True, to get the most out of it, its best to stay within Adobe, otherwise you need to encode to an intermediate anyway. He's just pointing out that if you can edit all in one space (which isn't too difficult with Premiere, After Effects, Encore, and countless plugins) there is never a need to encode until the final pass. Even a loss-less codec loses information any time the picture is changed and saved down.

Certainly that will not work for everyone, but will probably become more common as software and hardware gets better all around.

 

[andrechesini]

Proxy?

Proxy?

Hi,
Thanks everyone for the awesome discussion.
I could seens a bit off but I would like to know opnions about the PROXY workflow.
I had tried to edit without transcoding and the solution for my machine was to do it with proxy.
I had no problens doing it in PPCS4. And sound a interest solution for now.
Any coments?
Regards