HDMI Capture Problem SOLVED - AviSynth RULES!

AUDIO ISSUES

As you know, the GH2 does not send audio out the HDMI port. So, you need to record the audio either in-camera or to an external audio recorder and sync it up in post. However, there is a third option. Most HDMI recording devices have input ports for analog audio, giving you the ability to record audio along with the picture. The Blackmagic Intensity has such ports, and I've been experimenting with them.

The main problem you're going to encounter is that the GH2's picture coming out the HDMI port is delayed about 1/2 second. As a result, if you record audio through your capture device, it will be running about 1/2 second in front of the picture. Of course, you can fix this in post by sliding the audio track to match the picture. Or you can delay the audio before it hits the capture device. This is what I have been doing.

For these experiments, I routed the sound through an audio program where I introduced a delay, and then sent it on to the Intensity card. It took a little trial and error to find the right delay, but after I got it, I was able to record picture and audio in sync into the Intensity card. After working with all this silent footage, there was something magical about hearing sound along with this beautiful uncompressed picture. This is how people must have felt when they first heard sound movies!

Anyway, the thing I was concerned about was whether the sound would remain in sync after I ran the footage through the Avisynth script. Good news, it did!

So, now we have the third option for recording sound.

Does anyone know of an audio delay box that runs off batteries?
 
Last edited:
Ralph B said:
Does anyone know of an audio delay box that runs off batteries?
Click to expand...

I'm sure there's something out there. There are some simpler delay boxes that run off DC with an AC adapter, I'm sure some are MacGyverable to run off some DC batteries. For a probably similar pricepoint, some cheaper portable audio recorders with simple mic-ins offer a delay effect that could be outputted back out to an HDMI recorder. At that point though, you might just want to shell out for a more robust recorder like the H4n, so you'll have something to plug XLR mics into, with phantom power, effects, etc. if needed. At that point you could just use the audio from the recorder and sync up later, but if that's not conducive to your workflow, you can, again, turn on the delay effect and reoutput the audio to the HDMI recorder via the headphone/line-out... plug you can still record on the device if you want and have a backup.

I'd note though that since you have to run the HDMI footage through Avisynth anyway, you could apply the audio delay with AVISynth. Load the footage with audio, then use the delay() filter. Do you know the exact ms of the delay?
 
This is pretty exciting. I'm considering putting a PC together with an Intensity card for use in a studio (green screen mostly). Ralph, can you upload a video of the end result in Prores or some other efficient-yet-high-quality codec? The UT codec doesn't work on macs.
 
Ralph B said:
AUDIO ISSUES

As you know, the GH2 does not send audio out the HDMI port. So, you need to record the audio either in-camera or to an external audio recorder and sync it up in post. However, there is a third option. Most HDMI recording devices have input ports for analog audio, giving you the ability to record audio along with the picture. The Blackmagic Intensity has such ports, and I've been experimenting with them.

The main problem you're going to encounter is that the GH2's picture coming out the HDMI port is delayed about 1/2 second. As a result, if you record audio through your capture device, it will be running about 1/2 second in front of the picture. Of course, you can fix this in post by sliding the audio track to match the picture. Or you can delay the audio before it hits the capture device. This is what I have been doing.

For these experiments, I routed the sound through an audio program where I introduced a delay, and then sent it on to the Intensity card. It took a little trial and error to find the right delay, but after I got it, I was able to record picture and audio in sync into the Intensity card. After working with all this silent footage, there was something magical about hearing sound along with this beautiful uncompressed picture. This is how people must have felt when they first heard sound movies!

Anyway, the thing I was concerned about was whether the sound would remain in sync after I ran the footage through the Avisynth script. Good news, it did!

So, now we have the third option for recording sound.

Does anyone know of an audio delay box that runs off batteries?
Click to expand...


If you have time, I would be really grateful if you can send us some frame-grab of the footage above in tif or png. It would really make my day :Drogar-BigGrin(DBG) I understand that footage is going to be enormous.
 
REALITY CHECK

When I first conceived of using Avisynth to fix the GH2's HDMI output, it was my fondest hope that I could create a script that would be a no brainer to use. Just plop your files in a folder, press a button and be done with it. Sadly, that is turning out not to be possible. The problem is the threshold setting in fdecimate. There simply is no single number that will work on all clips. Some clips need the threshold to be set at 1.0, others as high as 4.0. The good news is there IS a setting that will work on each clip. The bad news is you may have to take the time to find it, by turning on metrics and scanning through the clip.

And athough I have not encountered this myself, it's conceivable that if a clip has radical changes in lighting and motion, the different sections may require different thresholds.

So, is this viable in a production environment? I can't answer that. All I can say is it sure is going to slow things down.

I don't want to scare you off, but like I say, this is a reality check. Know what you're getting into if you decide to go down this path.

Well, that's how things stand today. Who knows what tomorrow will bring.


EDIT: Disregard all of the above. Subsequent work has shown the cause of the problem was that I was recording to a 4.2.0 codec. If you record to a 4.2.2 codec, the script works beautifully, and you can set the threshold quite low (under 1.0).
 
Last edited:
BACK TO WORK

Here's a problem that needs to be solved. In the problematic red channel, TFM is not always able to completely eliminate the combing.



This is a duplicate frame that needs to be decimated. But, the residual red 'ghosts' force us to raise the threshold to 3.8 in fdecimate in order to do it. This is a very high number and not desirable because it will also decimate legitimate frames that have tiny bits of motion in them. Ideally the threshold should be as low as possible and certainly not greater than 2.

I've worked with the advanced settings in TFM in an attempt to get rid of the red ghosts, but nothing seems to have any effect. I believe if this problem can be solved, we very well might be able to set a threshold that will work for all footage.

If anyone wants to work on this, here is a link to the raw footage. It's recorded with UT 4.2.0.

http://www.sendspace.com/file/3b1jk9


I know several of you have been asking for still images, so here's an uncompressed frame grab from the above scene. This was shot at iso 3200, so there's a fair amount of noise.

http://www.sendspace.com/file/6p0x3y
 
Last edited:
Ralph - I don't think this latest clip can be "solved" , not even with a manual IVTC

When you step through the individual fields, there are some moments in time which are never recorded "cleanly", at least with respect to CbCr . So it's not a matter of decimation or field matching

But this only affects Cb and Cr - the luminance Y' is recorded correctly in that clip - so you could get a greyscale clip manually (or automatically) IVTCed

If you want to step through Y' fields
AssumeTFF()
SeparateFields()
GreyScale()


If you want to step through Cb fields
AssumeTFF()
SeparateFields()
UtoY()


If you want to step through Cr fields
AssumeTFF()
SeparateFields()
VtoY()
 
Last edited:
Do not worry too much; the times when the hdmi difference will be really pronounced is when there is significant motion and in particular - there are small particles that exist in a unique spot each frame (think Saving Private Ryan with the high shutter dirt in the air). It has to do with the way the Avchd processor in the gh2 'thinks' about continuity of motion. As long as there aren't such flash changes, color shifting can be fully color corrected and the detail will not appear compromised. (also do light the frame, the encoder is harsher to shadow data, using nostalgic helps this cause) One positive of the processor is it helps to lower noise.
 
Last edited:
PDR said:
Ralph - I don't think this latest clip can be "solved" , not even with a manual IVTC

When you step through the individual fields, there are some moments in time which are never recorded "cleanly", at least with respect to CbCr . So it's not a matter of decimation or field matching

But this only affects Cb and Cr - the luminance Y' is recorded correctly in that clip - so you could get a greyscale clip manually (or automatically) IVTCed

If you want to step through Y' fields
AssumeTFF()
SeparateFields()
GreyScale()


If you want to step through Cb fields
AssumeTFF()
SeparateFields()
UtoY()


If you want to step through Cr fields
AssumeTFF()
SeparateFields()
VtoY()
Click to expand...

Good work, PDR. Wow, that's unbievable - on the combed frames, there's actually two consecutive fields superimposed into one field for the chroma channels!

The only thing I can think of that might have some bearing on this is that I shot it with a shutter speed of 1/25. I'll shoot a similar scene tonight at 1/50 and 1/60 and see whether that makes a difference.

Do you think there's a way to reconstruct the chroma fields after TFM and before they hit fdecimate?
 
Last edited:
Sage said:
Do not worry to much; the times when the hdmi difference will be really pronounced is when there is significant motion and in particular - there are small particles that exist in a unique spot each frame (think Saving Private Ryan with the high shutter dirt in the air). It has to do with the way the Avchd processor in the gh2 'thinks' about continuity of motion. As long as there aren't such flash changes, color shifting can be fully color corrected and the detail will not appear compromised. (also do light the frame, the encoder is harsher to shadow data, using nostalgic helps this cause) One positive of the processor is it helps to lower noise.
Click to expand...

Thanks for your input. The goal at this point is to reliably get rid of the dupicate frames in the HDMI signal, in all situations. Making pretty pictures, we'll save for another day.
 
Ralph B said:
Thanks for your input. The goal at this point is to reliably get rid of the dupicate frames in the HDMI signal, in all situations. Making pretty pictures, we'll save for another day.
Click to expand...

Thanks for doing all this work Ralph. I'm hoping VK can hack this camera to fix all of these loops that Panny has set for us to jump through.
 
Ralph B said:
Do you think there's a way to reconstruct the chroma fields after TFM and before they hit fdecimate?
Click to expand...



There is a way if a fewer blends occur; but there are too many contiguous blends in Cb and Cr in that sample to do a successful reconstruction. There are field blend restoration filters you can use in avisynth (e.g. srestore), or even interploation filters like mflowinter() . So what you could do is reconstruct Cb and Cr then merge them back with Y' . To be able to do that successfully , you need a higher frequency of "good" fields interspersed between the bad ones

Manual IVTC is the "gold standard" - this is selecting fields and decimation by using your eyes; if you can't do it manually by examining fields, there is no combination of avisynth filters that will be able to do it "automatically"

The only thing I can think of that might have some bearing on this is that I shot it with a shutter speed of 1/25. I'll shoot a similar scene tonight at 1/50 and 1/60 and see whether that makes a difference.
Click to expand...

I doubt it will make a difference, but it's worth a shot . I assume the simultaneous avchd recording is ok in terms of no "ghosting"?

I suspect the problem lies where the RGB data (from the sensor) is converted to Y'CbCr signal in the hdmi ; somewhere along that chain. From your previous tests recording RGB, it appears the signal sent is Y'CbCr 4:2:2 (even though HDMI spec can carry RGB4:4:4 or even Y'CbCr 4:4:4 (or YV24) ) , so you would have to somehow access upstream of HDMI. There must be a disconnect somehere where the conversion from RGB=>Y'CbCr screws up CbCr but leaves Y' intact
 
Last edited:
Sage said:
I'm hoping VK can hack this camera to fix all of these loops that Panny has set for us to jump through.
Click to expand...

Yes, you , me and everyone! But Vitaliy has repeatedly said that a hack isn't coming anytime soon. So my feeling is, let's do what we can now.
 
Chroma issues and such

Chroma issues and such

On the issue of chroma, I've been quietly tinkering with AVISynth, trying to get the cleanest possible chroma on the chroma test chart footage that Barry put up. I've done alot of minor trial and error corrections to the chroma informations to optimize at least for a conversion to RGB so I can use ImageWriter() to look at still frames. I haven't come up with anything significant yet, other then correcting for the chroma shift inherit in the workflow and pretty much found the long way around doing FixBrokenChromaUpsampling() to get pretty much the same results. Haven't had the time to do much in the last couple of days, but I thought I might just post my long disgusting page of code I'm playing with, should anyone find any of the elements in it useful. Again keep in mind I've used this stricktly on Barry's chart footage with the end purpose of comparing stills using ImageWriter().

Code: 
MPEG2Source("C:\Users\Mick\Desktop\GH2-Wringer.d2v",idct=6)
 
# ConverttoRGB()
# ShowGreen("RGB32")
# ConvertBackToYUY2()
 
 
 
AssumeTFF()
 
TFM()
TDecimate(mode=1)
AssumeFPS(24000,1001)
#FixBrokenChromaUpsampling()
ConvertToYV24(chromaresample="spline36")
Y = Tweak(sat=0, coring=false)
U = UToY()
V = VToY()
 
# ---------- U Fields
UF = SelectEven(SeparateFields(AssumeTFF(U)))
LF = SelectOdd(SeparateFields(AssumeTFF(U)))
  ONE = SelectEven(SeparateFields(AssumeTFF(AssumeFrameBased(UF))))
  TWO = SelectEven(SeparateFields(AssumeTFF(AssumeFrameBased(LF))))
THREE = SelectOdd(SeparateFields(AssumeTFF(AssumeFrameBased(UF))))
 FOUR = SelectOdd(SeparateFields(AssumeTFF(AssumeFrameBased(LF))))
  ONE = ONE.Spline36Resize(one.width,one.height,0,0)
  TWO = TWO.Spline36Resize(two.width,two.height,0,0)
THREE = THREE.Spline36Resize(three.width,three.height,0,0)
 FOUR = Four.Spline36Resize(four.width,four.height,0,0)
 
UF = Weave(AssumeTFF(AssumeFieldBased(Interleave(ONE,TWO))))
LF = Weave(AssumeTFF(AssumeFieldBased(Interleave(THREE,FOUR))))
U = Weave(AssumeTFF(AssumeFieldBased(Interleave(UF,LF))))
 
# ---------- V Fields
UF = SelectEven(SeparateFields(AssumeTFF(V)))
LF = SelectOdd(SeparateFields(AssumeTFF(V)))
  ONE = SelectEven(SeparateFields(AssumeTFF(AssumeFrameBased(UF))))
  TWO = SelectEven(SeparateFields(AssumeTFF(AssumeFrameBased(LF))))
THREE = SelectOdd(SeparateFields(AssumeTFF(AssumeFrameBased(UF))))
 FOUR = SelectOdd(SeparateFields(AssumeTFF(AssumeFrameBased(LF))))
  ONE = ONE.Spline36Resize(one.width,one.height,0,0)
  ONE = ONE.Spline36Resize(one.width,one.height,0,0)
  TWO = TWO.Spline36Resize(two.width,two.height,0,0)
THREE = THREE.Spline36Resize(three.width,three.height,0,0)
 FOUR = Four.Spline36Resize(four.width,four.height,0,0)
 
UF = Weave(AssumeTFF(AssumeFieldBased(Interleave(ONE,TWO))))
LF = Weave(AssumeTFF(AssumeFieldBased(Interleave(THREE,FOUR))))
V = Weave(AssumeTFF(AssumeFieldBased(Interleave(UF,LF))))
V = V.Spline36Resize(V.width,V.height,0,0)
 
# ---------- Y Fields
UF = SelectEven(SeparateFields(AssumeTFF(Y)))
LF = SelectOdd(SeparateFields(AssumeTFF(Y)))
  ONE = SelectEven(SeparateFields(AssumeTFF(AssumeFrameBased(UF))))
  TWO = SelectEven(SeparateFields(AssumeTFF(AssumeFrameBased(LF))))
THREE = SelectOdd(SeparateFields(AssumeTFF(AssumeFrameBased(UF))))
 FOUR = SelectOdd(SeparateFields(AssumeTFF(AssumeFrameBased(LF))))
  ONE = ONE.Spline36Resize(one.width,one.height,0,0)
  TWO = TWO.Spline36Resize(two.width,two.height,0,0)
THREE = THREE.Spline36Resize(three.width,three.height,0,0)
 FOUR = Four.Spline36Resize(four.width,four.height,0,0)
 
UF = Weave(AssumeTFF(AssumeFieldBased(Interleave(ONE,THREE))))
LF = Weave(AssumeTFF(AssumeFieldBased(Interleave(TWO,FOUR))))
Y = Weave(AssumeTFF(AssumeFieldBased(Interleave(UF,LF))))
 
 
# ------ Chroma Adjust
 
U = U.Spline36Resize(U.width,u.height,0,0)
U = U.Spline36Resize(U.width,u.height,0,0)
V = V.Spline36Resize(V.width,u.height,0,0)
V = V.Spline36Resize(V.width,u.height,0,0)
U = U.Spline36Resize(u.width,u.height,0.5,-1)
V = V.Spline36Resize(v.width,v.height,0,-1)
Y = Y.Spline36Resize(y.width,y.height,0,0)
YToUV(U,V,Y)
 
 
# Spline36Resize(1920,1080)
 
ConvertToYuY2(chromaresample="Spline36")
ConvertToYV12(chromaresample="Spline36")
#u.Spline36Resize(u.width,u.height,0,-1),v.Spline36Resize(v.width,v.height,0,0)
 
ConvertToRGB(chromaresample="spline36")
 
ImageWriter("C:\Users\Mick\Desktop\Stills\000YUVShiftResize_00.bmp",start=33,end=33,type="bmp")
#ImageWriter("C:\Users\Mick\Desktop\Stills\0000GreenFix_Test.bmp",start=333,end=333,type="bmp")
 
Info()

Essentially there's code in there that breaks up the footage into each YUV channel, breaks those channels into 2 and 4 fields (which is what the manual method of FixBrokenChromaUpsampling() requires), then you can scale, shift, or swap those fields... then the code recombines those fields back to their respective channels... then you can scale or shift those channels before the code recombines them into regular YUY2/YV12 video before then switching over to RGB to use ImageWriter().

Not sure what state I left the current version of the code in... it gets a decent still although I'm not sure I left it at it's current best. And yes, at the moment alot of the code may be very redundant or whatever.. for instance some of the code pretty much does ConvertToYV12() just very manually..... again, have been sticktly playing/testing.

All of this was tested for use entirely after all the decimation and so forth... haven't gone in to whether it's perhaps useful within the TFM() TDecimate() code... i.e working on individual channels or whatever.
 
Another bizzare observation about that clip, is that neither Cb or Cr "bob" up and down when you step through

with *any* field examination there should be a slight up & down motion between consecutive fields because of the field offset. Even a static tripod shot should exhibit this up & down between fields (and it's normal and to be expected). You should be able to see this when you examine Y' fields , Cb , Cr or even Y'CbCr fields (or any other footage from any camera). You can see this on your other static footage uploaded with the red ball and green pen or Barry's Wringer test clip
 
What about to post some material of this thread in Doom9 forums where Avisynth gurus can resolve better this, or maybe write a specific plugin to resolve the downconversion of the chroma?
 
PDR said:
Another bizzare observation about that clip, is that neither Cb or Cr "bob" up and down when you step through

with *any* field examination there should be a slight up & down motion between consecutive fields because of the field offset. Even a static tripod shot should exhibit this up & down between fields (and it's normal and to be expected). You should be able to see this when you examine Y' fields , Cb , Cr or even Y'CbCr fields (or any other footage from any camera). You can see this on your other static footage uploaded with the red ball and green pen or Barry's Wringer test clip
Click to expand...

If the HDMI is blending sequential chroma fields together (and apparently it is), it makes sense that you wouldn't see a bob.
And it appears that the two chroma fields it blends together come not from the same frame, but from adjacent frames.
 
Last edited:
improser said:
What about to post some material of this thread in Doom9 forums where Avisynth gurus can resolve better this, or maybe write a specific plugin to resolve the downconversion of the chroma?
Click to expand...

I was eventually thinking of writing a custom function into an avisi file for the chroma correction, once optimized as far a possible, although perhaps with further work a singular custom function or plugin can be made to handle the whole process, if an automated way of dealing with all clips can be figured. Even if not... it could be a function with an argument to adapt to a few kind of the variations that have been seen.

Something like GH2HDMIFix(mode=blah).
 
Ralph B said:
If the HDMI is blending sequential chroma fields together (and apparently it is), it makes sense that you wouldn't see a bob.
Click to expand...

Yes, you might expect that; but if you look at the "normal" fields , where there isn't CbCr field blending, there should be the slight "bob"....and there isn't
 
You must log in or register to reply here.
Back
Top