AVCHD Lite 720p25 in 50p. How to remove 50p wrapper?

[dizpark]

Hi!

This is my first post here and I apologize, as it is not strictly GH1 related. But this forum seems like the perfect place to ask - with all gh1 hackers and stream specialists around.

Panasonic pocket digicams (like DMC-TZ10 here, but many others) record 25p stream, but save it in a 50p wrapper (or 30p in 60p in NTSC models). My understanding is that the camera records at 25p, but basically says it is 50p and additionally saves a bitstream flag to tell the playback device (or NLE) to play each frame twice.

Is there a way to remove this 50p wrapper and and "rewrap" the original stream back into 25p MTS container. In theory this should be possible, but perhaps something in the way how MTS file is "constructed" does not allow this (as 720 25p /30p is not a legal AVCHD format).

 

[GrahamH]

Yes there are ways, some of which involve transcoding to a non-AVC format (e.g. a Cineform workflow). So the first question is: how important is it to you to rewrap back to mts? What's your final delivery format going to be?

 

[dizpark]

I should have been more specific, but I did not want to bother you with details. The story is quite long.

It is "sort of important" to wrap back into MTS. I am setting up a workflow to edit DMC-TZ10 files in Womble Video Wizard on a rather old and slow computer. Womble is a pre-requisite at the moment. I am setting it up for my father, who is a) used to the simple interface of Womble and b) even if we upgraded to faster machine, any other NLE (like Sony Vegas) would be too much of a learning curve for him.

Neither that PC nor Womble can cope with raw MTS files, so I a looking at ways to transcode MTS into a format that into Womble can handle. It has to be high bitrate (no to lose IQ), I-frame only (editability) and preferably MPEG2 stream that Womble excels in.

I have tried and tested a wide variety of transcoding options - Cineform, DNxHD, Matrox I-frame, VoltaicHD, Elecard, AVCHD Upshift... From all of these only AVCHD Upshift produced files that can be edited relialbly in Womble. Some of the other solutions produced files that are not playable in Womble even with all the required codes installed. Some (like Cineform) are playable but transitions/effects cannot be applied etc.

So as I said AVCHD Upshift looks like the only solution I have found so far. It produces high bitrate I-frame only M2T files that are a joy to edit in Womble. Unfortunately in its latest version Upshift gets it "wrong" with the double frame 50P wrapper and produces transcodes with the wrong fps indication and wrong frame count. Upshift got it right in its previous versions, but there were some other problems there. So my idea is - if I presented Upshift with pure 25p stream in a 25p container, it would not get "lost". The workflow would look like this (convoluted, but workable)

Original footage -> rewrap in 25p -> Upshift M2t -> edit in womble.

So based on the above, it is important to wrap back into MTS. I am really constrained by what Womble can and cannot handle.

Yes there are ways, some of which involve transcoding to a non-AVC format (e.g. a Cineform workflow). So the first question is: how important is it to you to rewrap back to mts? What's your final delivery format going to be?

 

[svecher]

Are you sure it is a 50p wrapper and not 50i?

A useful tool that works on transport-stream level is Avidemux. I've used it for manipulating GH13 files.

 

[GrahamH]

Procoder should be able to ingest AVCHD Lite (i.e. MPEG4) and output 25P MPEG2 - not sure if it can do I-frame MPEG2 however.

Also, regarding files that Womble will accept, keep in mind there are MPEG2 transport streams (.MTS) and program streams (.MPG) and the two can be losslessly muxed back and forwards with freeware like TSRemuxer and others. If Womble won't take one version, maybe it would take the other?

Good luck!

 

[dizpark]

Even if it works, Procoder is way out of my budget. Incidentally I have also tried some open source MPEG2 encoders using available GUIs - as far as I am aware, they do not offer high bitrates that I require.

Procoder should be able to ingest AVCHD Lite (i.e. MPEG4) and output 25P MPEG2 - not sure if it can do I-frame MPEG2 however.

That PC will fold under the workload of AVCHD decoding. So it needs to be transcoded into something more editable, for sure. But thanks for the pointer about muxing stuff in different streams.

Also, regarding files that Womble will accept, keep in mind there are MPEG2 transport streams (.MTS) and program streams (.MPG) and the two can be losslessly muxed back and forwards with freeware like TSRemuxer and others. If Womble won't take one version, maybe it would take the other?

Good luck!

 

[GrahamH]

Just to be clear, my comment about MTS vs MPG was in the context of MPEG2, which as you say is Womble's preferred mode. HDV, for example, is an MPEG2 transport stream. Whatever workflow you use, your initial MPEG4 AVCHDLite footage is going to need to get transcoded.

 

[dizpark]

I am sure it is 50p wrapper. Here:

http://en.wikipedia.org/wiki/AVCHD#AVCHD_Lite


I cannot say I am very nimble with avidemux and tsmuxer, but I am not sure that these otherwise excellent tools can help me here.

I can use tsmuxer to extract video stream from the MTS file and I 'think' I can change the frame rate, but these tools do not give me access to "frame doubling" flag in the stream - that remains unchanged in the demuxed stream and creates problems further down the road.

So far I seem to have hit the dead end with this. I now realize that to accomplish what I had in mind I need a tool that can manipulate info *inside* the stream itself - specifically frame rate indication (tsmuxer can do this?) and the "frame doubling" flag (neither avidemux, nor tsmuxer give acces to this).

Anyway, thanks for all the help. I have definitely learned something new doing this.

Are you sure it is a 50p wrapper and not 50i?

A useful tool that works on transport-stream level is Avidemux. I've used it for manipulating GH13 files.

 

[PDR]

Incidentally I have also tried some open source MPEG2 encoders using available GUIs - as far as I am aware, they do not offer high bitrates that I require.

Yes some offer high bitrates; ffmpeg (free & open source) , hcenc (free, not opensource) can both encode to MPEG2 I-frame using very high bitrates. HCenc offers better quality but at >100Mb/s you 're not going to see the difference, and AVCHD Lite footage is pretty low quality to begin with

However, your problem is decoding, not encoding.

AVCHD Lite has a frame repeat flag that very few decoders acknowledge properly, hence the problems with incorrect durations and frame doubling.

As GrahamH pointed out, stripping a flag is pointless in your case, since you are going to re-encode to MPEG2 anyway. You just want to decode ignoring the frame repeat flag.

One way to decode properly is to use FFMpegSource2 through avisynth, frameserving into any encoder (in your case, for MPEG2 encoding, something like ffmpeg would be easy to do batch scripting and batch encoding). It handles the frame repeat flag properly, so you get 25p (or 29.97p for NTSC models)

Another option is to use a NLE like vegas, set it up in a 25p sequence, disable resampling , then either export directly to MPEG2 Intra (can't recall off hand if it has fine controls for MPEG2 encoding or limited bitrates) or uncompressed to encode with ffmpeg or hcenc. Vegas does handle the repeat flag properly if it is setup correctly

I don't have this camera, but I have tested footage out from it, so I know both methods will work

 

[dizpark]

First of all thank you for your great post.

Yes some offer high bitrates; ffmpeg (free & open source) , hcenc (free, not opensource) can both encode to MPEG2 I-frame using very high bitrates. HCenc offers better quality but at >100Mb/s you 're not going to see the difference, and AVCHD Lite footage is pretty low quality to begin with

Yes, I though of this route as well - abandon Upshift and go for free encoders. But I have to admit that tackling ffmpeg or hcenc is a bit daunting. I read through ffmpeg documentation and could not find reference to max bitrates offered, so I did not explore that further. But this is definitely an option.

And yes - based on my experiences with Upshift - something in the range of 50 Mbps is already perfectly sufficient for 17 Mbps AVCHD Lite footage.

One way to decode properly is to use FFMpegSource2 through avisynth, frameserving into any encoder (in your case, for MPEG2 encoding, something like ffmpeg would be easy to do batch scripting and batch encoding). It handles the frame repeat flag properly, so you get 25p (or 29.97p for NTSC models)

Can you guide me to webpage(s) that would describe how to set this up? This sounds very promising, but I have next to no experience with avisynth scripts, frameserving and batch encoding.

Another option is to use a NLE like vegas, set it up in a 25p sequence, disable resampling , then either export directly to MPEG2 Intra (can't recall off hand if it has fine controls for MPEG2 encoding or limited bitrates) or uncompressed to encode with ffmpeg or hcenc. Vegas does handle the repeat flag properly if it is setup correctly

I have access to Sony Vegas 10 studio HD, I don't think its MPEG2 encoder offers what I want. But I can export to Lagarith AVI file (or see if I can frameserve) and through a simple AVS script encode it in HCENC. I tested it on one file and it appears to work (Womble likes it. I need to demux the audio stream from the original MTS and remux it back together with HCENC output. If that does not through up some audio sync surprises this is definitely a good option (I would need to learn how to write batch files for this though).

In HCENC I changed the following settings (from default)

Average bitrate 50000
Bitrate Max 50000
Autogop off
GOP length 1
B frames 0
Interlacing progressive (van be left at autodetect, I guess)

The other settings I left at default values (since I do not know what they are doing). I will further experiment with bitrates, maybe good results can be achieved even with lower bitrates, but apart from that - do the above look like good values for high bitrate I-frame only intermediate encoding? Or something else should be changed?

 

[PDR]

It sounds like you're leaning towards the Vegas + HCEnc method?

For Vegas Pro (I haven't tested on Studio), you have to disable resample for each clip (or you'll get ghosting) , set the project settings to 1280x720p25, and export the same

You can create batch scripts using templates with avs batch scripter (there are several related tools that go by similar names) , and you can batch encode with HCBatchGUI . Here is a guide
http://forum.videohelp.com/threads/272050-How-to-Batch-Process-With-HCEnc

Note, HCEnc only encodes video, so you would still have to multiplex the audio. I would use the original audio to re-wrap into transport stream (instead of re-encoding audio & suffering generation loss). I would use TSMuxer , but I'm not familar with the CLI interface for it, so you might have to do each one manually

You might be able batch multiplex with imagompeg muxer, for mpeg2 program stream (.mpg) and it has a batch utility
http://homepage2.nifty.com/sophia0/freesoft/ImagoBatch.rar




For the ffmpeg method

1) Install avisynth

2) Download FFMpegSource2 and unzip & place in avisynth\plugins folder (you specifically need ffms2.dll and ffmsindex.exe)
http://code.google.com/p/ffmpegsource/

3) Open a text file in notepad and create your script, save, rename extension to .avs from .txt

e.g.

FFVideoSource("0001.MTS")

If you wanted to do audio & video (I don't recommend this in your case, because womble can smart render MPEG2, and I would remux the new elementary MPEG2 video .m2v with the old AC3 audio, so you don't get audio generation loss)

vid=FFVideoSource("0001.MTS")
aud=FFAudioSource("0001.MTS")
AudioDub(vid,aud)

4) You can use ffmpeg to encode to I-frame high bitrate MPEG2 , and even batch process. You can download autobuild binaries from here
http://ffmpeg.arrozcru.org/autobuilds/

The code for 1 video (if it's too high a bitrate for you, just increase the qscale parameter)

For your purposes, using a set bitrate is worse then using quantizer encoding. Complex scenes will get the bitrate they require for a certain level of quality, but low quality scenes will take up less bitrate. Furthermore, it will only be 1 pass (vs. 2passes required for accurate vbr). But if you wanted to you could use -b 50000k (or whatever bitrate)

ffmpeg -i input.avs -vcodec mpeg2video -qscale 1 -qmin 1 -intra -an output.m2v

If you had a directory full of .avs files you can batch encode by using a .bat file. Open notepad and write the following, rename .txt to .bat and double click it. ffmpeg.exe has to be in the same directory

for %%a in ("*.avs") do ffmpeg -i %%a -vcodec mpeg2video -qscale 1 -qmin 1 -intra -an %%~na.m2v

These just encode video, not audio. So would copy the original ac3 audio with tsmuxer (ie. re-wrap the new video with the old audio into a new .m2ts container, so there is no audio quality loss). You could modify the script to do audio as well (instead of -an and specify a container) , or use imagompeg muxer or the batch version of it as mentioned above

Or if you don't care about re-encoding the audio, you could do that with ffmpeg as well

You could also skip the extra step with vegas, and batch avs scripts with hcbatchgui instead of ffmpeg

 

[dizpark]

You are a goldmine, thank you very much.

It sounds like you're leaning towards the Vegas + HCEnc method?

For Vegas Pro (I haven't tested on Studio), you have to disable resample for each clip (or you'll get ghosting) , set the project settings to 1280x720p25, and export the same

Yes, it works the same in Studio. Disable resample is one of those Vegas "gotchas".

I would use the original audio to re-wrap into transport stream (instead of re-encoding audio & suffering generation loss). I would use TSMuxer , but I'm not familar with the CLI interface for it, so you might have to do each one manually

You might be able batch multiplex with imagompeg muxer, for mpeg2 program stream (.mpg) and it has a batch utility
http://homepage2.nifty.com/sophia0/freesoft/ImagoBatch.rar

Yes I intend to use the original audio. Tsmuxer is excellent to demux the original MTS to get AC3 stream. But I seem to have trouble muxing the m2v from HCENC and original AC3 back together into MPG container. Tsmuxer lets me mux them back into TS or m2ts. But I have yet to find a way to mux them into mpg. Imagomuxer gives me error 'not free space in inut buffer. Avidemux seems to offer streem copying into MPG, but despite my best efforts does not want to include the audio in the MPG file. All of this may or may not be a problem, because maybe Womble can handle those just TS or M2TS fine. I need to test this

The code for 1 video (if it's too high a bitrate for you, just increase the qscale parameter)

For your purposes, using a set bitrate is worse then using quantizer encoding. Complex scenes will get the bitrate they require for a certain level of quality, but low quality scenes will take up less bitrate. Furthermore, it will only be 1 pass (vs. 2passes required for accurate vbr). But if you wanted to you could use -b 50000k (or whatever bitrate)

I will test the other method that you describe as well. But I got alerted to your remark about preference of CQ over set bitrate. Doesn't this somehow apply to Hcenc encoding as well. In my test i used ABR at 50000 with 2 passes. But would a QC method be better for my intermediate codec uses? But at what settings - there are so many of them in QC in Hcenc (not just simple QC level)?

 

[PDR]

Yes I intend to use the original audio. Tsmuxer is excellent to demux the original MTS to get AC3 stream. But I seem to have trouble muxing the m2v from HCENC and original AC3 back together into MPG container. Tsmuxer lets me mux them back into TS or m2ts...maybe Womble can handle those just TS or M2TS fine. I need to test this

I think it can, if it doesn't I'll think of something else, you might be able to put into mpeg program stream (.mpg) with ffmpeg . HD streams are usually packaged into transport streams, so I should think tsmuxer + womble would work. Your original files came in transport stream, and other MPEG2 streams (e.g. HDV, MPEG2 blu-ray) are commonly packaged in transport streams

I will test the other method that you describe as well. But I got alerted to your remark about preference of CQ over set bitrate. Doesn't this somehow apply to Hcenc encoding as well. In my test i used ABR at 50000 with 2 passes. But would a QC method be better for my intermediate codec uses? But at what settings - there are so many of them in QC in Hcenc (not just simple QC level)?

Sure, you can use quantizer encoding there as well , I think it's on the 2nd tab if you're using the GUI. 1 pass instead of 2 is much faster as an added bonus . It wont make a big difference when you are using huge bitrates. I did a quick test with some AVCHD Lite footage and q=1 resulted in average bitrate of ~130Mb/s for a sports oriented sample. Also, lower bitrates are easier to edit than higher bitrates, so if you're findin it sluggish, you may have to back off

 

[dizpark]

Ok, I just want to give some feedback and share my experience with methods suggested by PDR.

1) FFMpegSource2/avisynth/ffmpeg solution. While I potentially would prefer this (because I would be able to automate more tasks with batch processing) , I have not had success. The encoded m2v has problems - it starts with garbled/pixelated section (fraction of a second), and then the video footage is for the lack of a better word "squeezed in or sped up", and there are blank frames at the end of the footage - if I watch the footage in vlc or any NLE, these final empty/black frames contain a message CAVIStreamSynth: system exception - Access Violation at 0x10012380, reading from 0x4c). The problem appears to be not with ffmpeg, rather with the way how video is served from FFVideoSource. If I use the Vegas generated Lagarith AVI as a source to feed Ffmpeg (through AVISourcē("input.avi") AVS), then the encode is succesful.

2) The other method - generate lagarith AVI from Vegas timeline then encode with Hcenc or ffmpeg works. At similar file sizes Hcenc produces visually superior results. However both Hcenc and ffmpeg are visually inferior to the Upshift results, even if Upshift uses lower bitrates (so the encoder that Upshift uses is quite good, shame about the framerate problem). The difference is not huge but visually noticable - there is loss of contrast, slight softening and slight color shift with Hcenc and Ffmpeg encodes. However in real life and for amateur use these are perfectly acceptable - with minimal post-processing these can be brought back to the original MTS "looks".

3) Also it looks like Womble quite "likes" TS muxed by tsMuxer. I was able to use effects and filters etc withou Womble acting up.


Once again, thanks for great inputs and suggestions.

 

[PDR]

1) FFMpegSource2/avisynth/ffmpeg solution. While I potentially would prefer this (because I would be able to automate more tasks with batch processing) , I have not had success. The encoded m2v has problems - it starts with garbled/pixelated section (fraction of a second), and then the video footage is for the lack of a better word "squeezed in or sped up", and there are blank frames at the end of the footage - if I watch the footage in vlc or any NLE, these final empty/black frames contain a message CAVIStreamSynth: system exception - Access Violation at 0x10012380, reading from 0x4c). The problem appears to be not with ffmpeg, rather with the way how video is served from FFVideoSource. If I use the Vegas generated Lagarith AVI as a source to feed Ffmpeg (through AVISourcē("input.avi") AVS), then the encode is succesful.

Do you have haali media splitter installed ? ffms2 requires external splitter for transport streams

Theoretically , ffms2 is the best way, because it avoids the RGB conversion through vegas . Input and output levels are identical.

I should mention several other people use this method successfully (that have this camera)

What NLE? vegas studio? Some have issues with editing elementary streams . If you wrap in .m2ts (or even export .m2ts with ffmpeg in 1 step) , NLE's shouldn't have any issue

ffms2 does have issues with clips that are not cut at frame boundaries cleanly. If you use the native files output from camera, there should be no issue. But if you cut the clips before (e.g. with tsmuxer or other software) , then you can get a few bad frames at the beginning

2) The other method - generate lagarith AVI from Vegas timeline then encode with Hcenc or ffmpeg works. At similar file sizes Hcenc produces visually superior results. However both Hcenc and ffmpeg are visually inferior to the Upshift results, even if Upshift uses lower bitrates (so the encoder that Upshift uses is quite good, shame about the framerate problem). The difference is not huge but visually noticable - there is loss of contrast, slight softening and slight color shift with Hcenc and Ffmpeg encodes. However in real life and for amateur use these are perfectly acceptable - with minimal post-processing these can be brought back to the original MTS "looks".

Likely a vegas levels issue , because it's converted to RGB through vegas, and back to YV12 . Vegas (especially the studio version) doesn't offer much control at all over YV12<=>RGB conversion

 

[dizpark]

Do you have haali media splitter installed ? ffms2 requires external splitter for transport streams

That was IT! With Haali installed, it now works perfectly and the quality is excellent. No color shifts, no loss of contrast, nothing... You were right, the quality is far, far better than through Vegas-Hcenc route.


I can run the encode from the command line, but I have a problem with the batch file

for %%a in ("*.avs") do ffmpeg -i %%a mpeg2video -qscale 1 -qmin 1 -intra -an %%~na.m2v

All the files - original MTS files, AVS, BAT and Ffmpeg.exe are in the same directory. I get the error message "Unable to find a suitable output format for 'mpeg2video'".

Here is the contents of the Command prompt

C:\Users\Didzis\Desktop\ffmpeg_method\batch_tests>ffmpeg -i 00013.avs mpeg2video
 -qscale 3 -qmin 1 -intra -an 00013.m2v
FFmpeg version SVN-r25658, Copyright (c) 2000-2010 the FFmpeg developers
  built on Nov  3 2010 04:12:55 with gcc 4.4.2
  configuration: --enable-gpl --enable-version3 --enable-libgsm --enable-pthread
s --enable-libvorbis --enable-libtheora --enable-libspeex --enable-libmp3lame --
enable-libopenjpeg --enable-libschroedinger --enable-libopencore_amrwb --enable-
libopencore_amrnb --enable-libvpx --disable-decoder=libvpx --arch=x86 --enable-r
untime-cpudetect --enable-libxvid --enable-libx264 --extra-libs='-lx264 -lpthrea
d' --enable-librtmp --extra-libs='-lrtmp -lpolarssl -lws2_32 -lwinmm' --target-o
s=mingw32 --enable-avisynth --cross-prefix=i686-mingw32- --cc='ccache i686-mingw
32-gcc' --enable-memalign-hack
  libavutil     50.32. 5 / 50.32. 5
  libavcore      0.12. 0 /  0.12. 0
  libavcodec    52.94. 3 / 52.94. 3
  libavformat   52.84. 0 / 52.84. 0
  libavdevice   52. 2. 2 / 52. 2. 2
  libavfilter    1.54. 0 /  1.54. 0
  libswscale     0.12. 0 /  0.12. 0
Input #0, avs, from '00013.avs':
  Duration: 00:00:09.80, start: 0.000000, bitrate: 0 kb/s
    Stream #0.0: Video: rawvideo, yuv420p, 1280x720, 276480 kb/s, 25 tbr, 25 tbn
, 25 tbc
Unable to find a suitable output format for 'mpeg2video'

I tried to google for this message but did not get wiser. Do you have any idea what is going on here? Many thanks.

 

[PDR]

Sorry about that, typo . It should say "-vcodec mpeg2video" . I have edited the earlier post as well. I should have mentioned to add "pause" to leave it open if there were any error messages

for %%a in ("*.avs") do ffmpeg -i %%a -vcodec mpeg2video -qscale 1 -qmin 1 -intra -an %%~na.m2v
 
pause

 

[dizpark]

Thank you. I should have spotted the discrepancy between command line and bat syntax myself.