how (not) to lose generations

S

spanky

Member
hi
i'm itchy over this generation-loss factor in video making. people tell me all kinds of things - if you rewind the tape, you lose a generation. if you don't rewind the tape, you lose a generation. if you shoot over existing footage, if you digitise the footage, if you shoot lp, you lose a generation.

so... any truth to the old wives tales?

cheers

spanky
 
Digital is digital. If you make a copy of it, you make a copy of it. The only thing that will degrade the image is losing bits (tape going bad could do it, likewise dirty heads), or rendering to a lossy format.

There really aren't "generations" in the digital realm in the same sense as there are in the analog.

Dan
 
GenJerDan said:
Digital is digital. If you make a copy of it, you make a copy of it. The only thing that will degrade the image is losing bits (tape going bad could do it, likewise dirty heads), or rendering to a lossy format.

There really aren't "generations" in the digital realm in the same sense as there are in the analog.

Dan
Click to expand...
Technically speaking - in DV you lose a generation every time you recompress. So, if you capture, edit (even if you don't apply any effects and use only straight cuts), print to tape, capture, ..., you are losing information. It's the same as opening and editing the same JPEG file all over.

It's not as bad as in analog, but there is generation loss. However, this doesn't have anything to do with rewinding or recording over a tape - that's a different kind of loss, and since I don't do it, I can't comment on how or why it affects the quality of a new recording.

Jerry
 
XCheck said:
Technically speaking - in DV you lose a generation every time you recompress. So, if you capture, edit (even if you don't apply any effects and use only straight cuts), print to tape, capture, ..., you are losing information.
Click to expand...
I don't believe this is correct, although it may vary with your NLE. I speak for Premiere Pro, but I think other NLEs more or less work the same.

If you are editing DV, and use only straight cuts -- no transitions and no titles -- the DV is never recompressed. It is written out identically to the way it was read in. Even if you export to tape, since it's a digital signal, you can read it back in exactly as it was written.

But I think the original question was really about reusing tapes. I personally don't do it . My favorite saying is "media is cheap, data is expensive."
 
Yes, if you use 24pA (or 30p or 60i), you can edit without any generation loss.

If you do any color correction, add any transitions, etc, however, you will lose a generation. Final Cut Pro (and most NLEs, I assume) is smart, and typically you won't lose more than a generation anywhere since it recompresses from the original footage every time you use a new effect.

Going between After Effects, Final Cut Pro, etc. etc. is where you really experience problems. Also, certain DV codecs deal with generation loss better than others. Typically the results aren't bad, but in projects with effects in them I've noticed some pretty serious issues.

Rewinding, recording over (bad idea, causes drop outs), etc., recording in LP mode (also a bad idea, drop outs), will not cause generational loss.
 
Right -- this is the difference between digital tape and analog tape. The term "generation loss", as applied to analog tape, comes into play whenever you copy the tape. Each copy is a "generation down", and would be of lesser quality than the original.

With digital tape that's not really an issue. With digital tape you make a direct clone, bit-for-bit identical, no generation loss. The only thing that could happen that would make a digital tape copy inferior to the original would be if a dropout occurred. Other than that it's identical.

Also, transferring to a computer and transferring back to tape is equally lossless (again, with the caveat that a dropout might occur). But other than that, you can transfer the data to your computer, chop it up, move it around, and then transfer it back, and it will be completely lossless -- no generation loss.

Note: this applies for intraframe compression systems, such as DV, DVCPRO-HD, MJPG, and MPEG-IMX. That does not apply for GOP-based compression systems such as MPEG-2, where any modification to the data will result in at least partial generation loss (more later*).

So now we've changed the meaning of "generation loss" -- the concept still exists, but it happens at a different place. Now instead of happening during copying, we apply that term to describe what happens when the video gets recompressed. Because DV is a "lossy" compression scheme, it loses data when it compresses (as does most every form of video compression). So if you modify the image at all (by color correcting, or adding a fade or dissolve, superimposing a title, anything like that) then the data will need to be recompressed before being sent out to tape. That recompression causes a "generation loss". Some compression systems handle that generation loss better than others; DVCPRO-HD is nearly lossless, DV is pretty good, HDV is not so good at it. Going a few generations (rendering a section out, re-importing it, modifying it, and rendering it out again, and repeating the process) is never good for your footage, multiple generations cause artifacts to build on top of artifacts and the footage can quickly fall apart. Again, some codecs handle it better than others (DVCPRO50 being among the best) but it's a process best avoided.

So editing DV on the timeline, if you were to do a primarily cuts-only edit, adding a few fades here and there, maybe a title, etc... when you render the final version back to tape, almost all of your program will be a direct clone of the original footage. The only portions that incur a generation loss would be the sections where you modified the footage (by adding titles or fades, etc). Those segments would be re-rendered, taking a generation loss in quality (although, depending on the quality of the codec being used, that may be fairly minimal, or it could be quite noticeable).

When rendering to something other than DV tape, the entire program will be recompressed (for example, when rendering out to MPEG-2, to make a DVD). This is not really a "generation loss", per se, because you're transcoding the entire program to a different codec. There will be loss involved, because MPEG-2 is also a lossy codec, but a) it's unavoidable, and b) everything will be treated equally (i.e., the fade or dissolve sections won't suffer two generation losses, they'll be transcoded from their original source).

* about long-form GOP editing: MPEG-2 editing involves an entirely different complication. Each frame doesn't stand on its own, but is dependent on the contents of the frames surrounding it. As specifically relates to HDV, frames are grouped into groups of either six (for 720p) or 15 (for 1080i) frames. Change one frame, and you'll have to re-render the entire group (and, possibly, the group preceding the affected group, and/or the group following, depending on the bidirectional frames). How the editing program handles this is apparently dependent on the editor.

If one edits MPEG-2 on a group boundary only, it's possible for the editor to be able to behave like a cuts-only DV edit. If you always cut exactly on a GOP (group-of-pictures) boundary, and made your cuts in exactly GOP-length sections, then it's possible the editing program could export back to MPEG-2 the same way you could export a DV timeline back to DV -- losslessly. It's much more restrictive to edit that way in MPEG-2, of course -- for example, in 1080i HDV, you'd be limited to only cutting on every 15th-frame, and you'd have to cut out at least half-a-second of footage (or as much as you want, as long as it is done in half-second increments). So while it's technically possible to edit in such a way as to output back to MPEG-2 without a generation loss, I don't think this is a practical workflow -- editing on GOP boundaries is very restrictive.

So more likely, one will be editing on whatever frame one thinks best. And that can cause problems. If the flow/sequence of GOPs is disrupted, re-rendering of the entire group is necessary. And, depending on how the editor is implemented, re-rendering of EVERY group is probably going to happen too. Cutting out one frame may necessitate re-rendering the entire timeline, causing a generation loss on the entire program.

For example -- groups are somewhat self-contained units of 15 frames each. Every group has a specific structure -- they start with an I-frame, and follow with a sequence of Predicted and Bidirectional frames (IBP sequence). Every frame is dependent on every other frame for its info. If you cut out a frame, you disrupt the sequence, and the group no longer "works" -- it has to be completely re-rendered. But here's the kicker -- if you disrupt the flow of a group, you likely disrupt the flow of every group after it on the timeline. Which means that changing one frame may necessitate re-rendering the entire timeline! And because MPEG-2 is a very lossy compression scheme, the generation loss will be quite noticeable.

I haven't heard of any other HDV editor even trying to avoid re-rendering; as far as I know, all native HDV editors just presume that if you're rendering out, you'll be recompressing the whole timeline. It's possible Pinnacle's Liquid 6.1 can avoid that worst-case scenario, by providing for intelligent group editing and re-ordering of the group sequences, using sophisticated MPEG splicing. I've ordered a demo disk to give it a try. But, even if it does, it's potentially problematic -- I discussed this with Adam Wilt, since he wrote up a big review on the state of HDV editing for DV magazine. He says that even if Pinnacle implements the ability to only re-render affected groups (and the affected group would include the one that got cut/dissolved/faded, and potentially the group before it which may have a bidirectional frame that references the affected group), the generational loss effect would mean that your footage could end up looking like an old bad tape splice in film, where the footage looks good for the most part but whenever there's an optical effect everything gets blurry for the duration of the effect, then snaps back into being "good" when the effect is over.

As a practical matter, it shouldn't really be much of an issue, because it really only becomes an issue if HDV users are re-rendering back out to HDV tape. If you're transcoding to another codec or another delivery medium, or re-rendering to DVD-compatible MPEG-2, then there won't be any effect of cumulative generation loss because the entire source will be transcoded to the new codec.
 
No kidding. Is there a way to be alerted every time Barry posts something? That I way could just go straight there!
 
Nice Post barry,

but then is also another possible way for NLE's to deal with MPEG2 streams and any GOP based system..

I don't know how Final Cut Pro 5 will work... but this is an assumption on a method that might be employed..

If you think for a moment, lets say you have a clip of HDV footage. You open that clip and set in an out points, then add it to your timeline.

lets say your in point is 1 frame from the next I-frame and your out point is 2 frames past an I-Frame. The NLE could conceivably decode the all the frames from the I-frame 14 frames before your in point, to 13 frames past your outpoint. So you'll have excess frames sitting in memory, that are just never shown. In this way, the NLE could intelligently drop those frames from memory and do a little shifting to properly play your timeline.

This may sound intensive, and it is. But the solution seems simple enough if you want lossless I-frame style DV Editing with HDV.

taking this a step further, FCP renders like all other NLE's to a scratch disk, so in theory the NLE could extract those frames from the GOP and plop them into a file on the scratch disc, therefore never needing to decode and go through that process again.

As far as I know, frame extraction using a method like this would not degrade the footage at all (other than what it has already been degraded to with MPEG2 compression).

An intelligent NLE can easily use HDV in much the same way as DV(on the outside, internal workings not withstanding). This is the programmer in me coming out again.

if I were to make a system it would be this, just purely based on timecode in/out points with a frame extraction method.

I can explain further if needed. I am pretty confident FCP 5.0 emplores something similar to what I have said, since it advertises native HDV editing like DV editing.

Any comments/thoughts barry?
 
Frame extraction doesn't cause a generation loss. It's when you render back out that you incur a generation loss.

Pinnacle solved the frame-extraction issue long ago, as far as editing goes. You can definitely edit natively, although editing long-form GOP natively can be much, much, much slower than editing in an intraframe codec. That's why so many HDV editing programs are looking to intermediate codecs (such as Apple's AIC, or CineForm's) -- it makes the timeline much more responsive.

The question posed in this thread wasn't how to speed up access to frames, it was how to avoid generation loss. I was trying to elucidate on how generation loss comes to happen in NLE operation -- but it only comes into play when you render out, not during normal editing (i.e., cutting, moving, fading/dissolving, etc). All those operations will still refer to the source footage in its first-generation form. If you pre-render, or render something out and re-import it, that's when you'll encounter the generation loss. Or when rendering out the timeline for your final version. However, like I said, if you're rendering to a different codec for delivery (such as to DVD) then there'll need to be a recompression no matter what codec you were using, so it's not a disadvantage to HDV in that case, all codecs would need to be recompressed.
 
Constantine said:
No kidding. Is there a way to be alerted every time Barry posts something? That I way could just go straight there!
Click to expand...
Yeah, something like 'subscribing' to a poster. I agree with you guys - any time I feel the urge to think I am smarter than the average bear, all I need to do is to read one of Barry's post and that will put me back on earth. This board is the best thing that happened to my ego. :engel017:
 
You must log in or register to reply here.
Back
Top