Potential Clip Problems, Please check...

[PKraft]

The problem is actually with Mac users. They can't use NTFS, so solid state cards intentionally screw the majority (PC users) by formatting their drives with FAT32 to accommodate the minority (Mac users).

Who says that a Mac cannot handle (read from/writeto) NTFS? It is better to get informed before to start bashing.

A very simple solution would be for solid state card makers to allow us to format the cards as we see fit (or drives in the case of Firestore). With NTFS (or HFS for Macs), long record runs would be a single file - problem solved.

I absolutely agree as for NTFS, HFS not necessary, see above.

Lacking that, different cards for PC and Mac users should be made available.

Not necessary, see above.

Bottom line is, all solid state devices/external drives should at a minimum allow full functionality for the vast majority, meaning NTFS formatting. If they still want to accommodate Mac users, fine. Just don't screw the majority in the process.

I absolutely agree as for NTFS, with a bottleneck of FAT 32 due to PC users though. FAT32 being the least common denominator of all platforms to be included.

(OK, that's off my chest. I'm better now.)

OK, that's off my chest. I'm better now. ;-)

 

[ajcourtney]

Well it would be nice if the hosting device gave the user the opportunity to format the card in whatever filesystem he wanted - probably impractical - but that is certainly not what is happening today. So, my previous question becomes all important:

Does Panasonic provide software to seamlessly reassemble these <4GB clips back into the original contiguous clip as displayed in cam? If the answer is "no" then I believe, based upon my past 10 months' of experience dealing with this AVCHD file size limit, the user will be entirely reliant upon his video editing software of choice to perform this function for him. And again, based upon past experience, you may be waiting for a very, very long time for this issue to be addressed, IF IT EVER gets addressed by your video editing software vendor.

jeff9329 writes: "Im having no camera problems/bugs and no editing problems at all in Vegas 8.0c."

Are you sure about that? Because I doubt that Vegas will seamlessly reassemble these clips back to the original file because it can't do it for the Canon cam's. So if the Panny files are a straight binary join like the Canons', I would expect to see 1-2 dropped frames and some dropped audio samples. The easiest way to test the audio would be to pipe in a live concert feed, set the cam on the table, press record, and walk away - which is exactly what I did with the Canon. When I brought the multiple files into Vegas (build 217), every splice point exhibited dropped audio - the rhythmic nature of continuous, live rock made it extremely easy to detect these splice points. So perhaps you could run a similar test and let us know what happens.

In any event, I would imagine Panasonic to come back and say use the DVCPro HD converter if you don't want any problems. Other than that, complain to your video editing software manufacturer because it's their problem, not ours. And that means you now have a mandatory conversion in order to avoid the issue, at least at this point in time.

 

[Barry_Green]

The AVCHD file system works. If there are problems, they're with the NLEs, and there's nothing Panasonic could or would do about that. The camera is hardware. NLEs are software. NLEs are easily fixable. Cameras not so easily fixable, and even if they were, there's not a problem.

You can play spanned AVCHD files back in the camera just fine; I've got several examples of spanned AVCHD files and they play great. You can play them back in the AVCCAM Viewer software, you can play them back in a blu-ray player, they work fine -- there is no stuttering or glitching or anything. You can play them back in a PS3, they work fine.

Any issues you folks are encountering are with the NLEs. It's easy to demonstrate that the 4GB files and the spanning structure work flawlessly.

Does Panasonic provide software to seamlessly reassemble these <4GB clips back into the original contiguous clip as displayed in cam?

Not sure what you're asking. I assume that by "<4GB" that you actually meant ">4GB", right? Because any clip less than 4GB should already be one contiguous file.

I think what you're asking here is: does Panasonic provide some software that will take the chunks of clips caused by the FAT32 file format's 4GB file size limitation, and export these spanned clips into NTFS or OSX or some other operating system's file format as a contiguous file that's greater than 4GB, right? So 10GB of source clips (being two 4GB clips and one 2GB clip on FAT32) would become a 10GB clip on NTFS, right? If that's the question being asked, then no, there's no software that does that as far as I know. Seems like a trivial piece of software to write up though; I'm sure some 3rd party could do it in about half an hour. Maybe ShotPut Pro could do it (if not in the current version, in a simple revision?)

In any event, I would imagine Panasonic to come back and say use the DVCPro HD converter if you don't want any problems.

Well, but see, that's kind of funny because the DVCPRO-HD converter uses the same 4GB file size limitation. So you'd end up with clips that were actually 4x as spanned (assuming 1080 clips). If you had 6GB of source footage, that's one spanned splice in your original AVCHD footage, but you'd have about 25GB of destination footage, making for six spanned splices of DVCPRO-HD footage. Yet we know that all NLEs handle spanned DVCPRO-HD just fine.

It's not the spanning that's a problem. That's a demonstrated and proven system. If there's an issue it's with the way the individual NLEs are handling it. Which means that yes, you'll need to get an update from your NLE manufacturer.

 

[Bassman2003]

Your correct.

I just spoke to tech support and the tech said he would look into it.

When I mentioned that Panasonic should release a little utility that you could pass the clips through to "glue" them back together, he crawfished a bit and did not want to go there.

From a consumer point of view, and this being called a pro cam, they better address this because I purchased this camera to be able to record for long periods of time!

I also would rather the material stay full raster and not have to go to DVCPro HD.

We will find out!

 

[ajcourtney]

AVCHD is not a filesystem. It's a format that uses outdated filesystems that have created this mess of incompatibilities that people have experienced, are experiencing, and will continue to experience. AVCHD's compression algorithm may be superior to MPEG-2, but its implementation thusfar has been a complete crapshoot for the consumer WRT these software inconsistencies. The ONLY vendor who has been able to sidestep this file size problem has been Sony because it provides the PMB (or whatever the acronym is) software designed to re-splice these <2GB (FAT) and <4GB (FAT32) clips back into the larger contiguous original "take" (defined as the moment in time record was pressed first to initiate recording and a second time to end the recording) upon transfer to the editing computer.

If the software is so easy to create, as you put it, tell that to Pixela - the software company Canon used to outsource this very task. They have chosen to impose their own limit of a one hour contiguous clip last time I spoke with their development team. The software must be smart enough to recognize that clips 1,2,3-5 belong to the same contiguous file; yet clips 6-9 are individual takes <2GB or <4GB (depending on FS). Unless the cam is providing some sort of metadata in the broken clips from the original contiguous take, how is the software smart enough to know which clip is part of a larger contiguous clip, and which is a separate take?

Trust me, this is a very complicated issue that has been discussed at great lengths since the first AVCHD cam's were released. Many of us have seen firsthand the games these manufacturers have played this year to absolve themselves of the responsibility to get the clip as it appears on cam into your editing computer PERFECTLY. You say it isn't their responsibility. I disagree and say it is.

This will prove to remain the single most important AVCHD topic in the nearterm for anyone requiring anything more than ~20 minutes or so of continuous footage.

 

[Barry_Green]

I don't know what you've been dealing with, but the 150 was designed to handle recording 12 HOURS of continuous recording, and it does it quite well.

And yes, there is metadata in the clips that specifically tells the computer how to stitch together HMC150 clips. I would assume, but don't know, that the HSC1U and HMC70 both have that capability as well.

I don't know what Canon or Pixela have done. If Canon's restricting themselves to an hour, that's their choice, but the HMC150 has no such restriction.

The file format (FAT32) was chosen because it's universally compatible. File formats that support files larger than 4GB are not universally compatible. FAT32 was the only reasonable choice.

As far as software to re-stitch it, as a retired professional computer programmer I do feel on rather safe ground to say that, at least as far as HMC150 footage goes, it would take someone about half an hour to write an app that stitched the files back together into a seamless, lossless 12-hour contiguous take of up to 32GB.

Heck, I wonder if the AVCCAM Writer software already does that? Maybe there's an option for it? I should try to dig into it.

You say it isn't their responsibility. I disagree and say it is.

It is their responsibility to provide a system that works. They have done that. You can play back 12-hour clips seamlessly. It is the NLE company's responsibility to make sure that their NLE can work with the footage as intended. But all the hardware already works exactly as intended.

 

[ajcourtney]

If the camera was the sole object as a means to an end, then yes, the system performs flawlessly. And that will be Panasonic's response - guaranteed. If, however, you're like 99.99999% of users, you will require the camera to interface with your chosen NLE. And this is the point of failure. Your opinion is it is the NLE's responsibility to correctly re-splice these broken clips back into the original contiguous file. It's my opinion that it's the camera manufacturer's responsibility to perform that operation since that party is the one that decided to use a filesystem that required the files to be divided up in the first place. Neither one of us is right or wrong. It's an opinion.

As it stands today, I will vote with my wallet by choosing not to purchase this cam until its AVCHD footage is universally supported by all NLE's advertising native AVCHD editing capabilities. How that feat is accomplished is largely irrelevant; but I would definitely prefer to see a solution provided by Panasonic since that would remove the mandate from the ISV's for supporting this product in future iterations of their software packages - something that few, if any, ISV's would ever commit to indefinitely.

BTW, as was already mentioned, Mac's running OS X and later (and possibly even later versions of OS 9) can read NTFS formatted partitions. And I believe some UNIX variants can do so as well. Not being able to write to that partition is irrelevant in this workflow model.

 

[Bassman2003]

As far as software to re-stitch it, as a retired professional computer programmer I do feel on rather safe ground to say that, at least as far as HMC150 footage goes, it would take someone about half an hour to write an app that stitched the files back together into a seamless, lossless 12-hour contiguous take of up to 32GB.

Hey Barry,

I am sure this will get resolved, but if it easy to write this kind of software, why doesn't Panasonic put out a small program like this for its customers?

We have edit what we shoot!

Is the NLE specific or all of the NLEs?

I know Edius is not spanning the clips, native or with CanopusHQ.

 

[jeff9329]

jeff9329 writes: "Im having no camera problems/bugs and no editing problems at all in Vegas 8.0c."

Are you sure about that? Because I doubt that Vegas will seamlessly reassemble these clips back to the original file because it can't do it for the Canon cam's. So if the Panny files are a straight binary join like the Canons', I would expect to see 1-2 dropped frames and some dropped audio samples. The easiest way to test the audio would be to pipe in a live concert feed, set the cam on the table, press record, and walk away - which is exactly what I did with the Canon. When I brought the multiple files into Vegas (build 217), every splice point exhibited dropped audio - the rhythmic nature of continuous, live rock made it extremely easy to detect these splice points. So perhaps you could run a similar test and let us know what happens.

AJ:

I am currently editing a long wedding project. It's a multicam (only 1 HMC150) edit and I have not rendered any video yet. When I let the only 4GB clip in this project run to the next clip in the NLE, I don't see a stutter. That's not a good test like a render will be though. I only have one of these long clips as I never just let the cameras run continuously. We shouldn't have to work around issues like this, but for my style of shooting, I almost never have any single clip of much length before a camera angle switch, so it won't affect me. As Barry said in a previous post, maybe some Pana software actually uses all that header info and reassembles the clips. I have not needed any software other than NLE, so I never even looked at the software provided with the camera.

Another note on the HMC150 editing; I ran the deshaker script on a bunch of HMC150 handheld clips and it worked flawlessly and the footage looked pretty lossless. Clip renders using the script were giving me about 4 FPS which is livable.

Jeff

 

[mcsmooth]

I don't remember the AVCCAM viewer stitching the files (not 100% on that), but as mentioned, there is no need for it to since the clip information references it properly as one clip. Any software (viewer, editor, etc) that reads the format properly will also show it as one clip, so it really should have no impact unless you are trying to copy stream files by themselves (not the best idea for most purposes).

Agreed that FAT32 is a necessary choice for right now since it works with everything. It makes sense for programs NOT to automatically stitch the files in case you wanted to move the them back to the card and stay within AVCHD's specs. I can see people wanting that feature to be able to convert to a single stream file though... If programs do not already do this, expect tools to come about soon since this has been done with other forms of mpeg for a long time and does not require much work. Anything that smart renders would be able to do this too.

Look at most any movie's file structure on DVD and you'll see the streams are split into 1GB files for any video bigger than a 1GB (which most movies will have for the feature). This is a similar limitation that works the same way... every player (or software) that follows standards will see and play these back as one clip without skipping a frame.

 

[mcsmooth]

Hey Barry,

I am sure this will get resolved, but if it easy to write this kind of software, why doesn't Panasonic put out a small program like this for its customers?

We have edit what we shoot!

Is the NLE specific or all of the NLEs?

I know Edius is not spanning the clips, native or with CanopusHQ.

Not just to you... but to all that are waiting on Panasonic, sure it would be a nice gesture, but I wouldn't hold your breath. Why should they be required to fix a problem with someone else's software? This IS NLE specific! The camera follows the AVCHD spec, so it is up to the software to follow suit. There are many posts in here of other software that does not have this problem.

On that note, I love this camera, but I would not recommend it (or ANY other camera with a new codec) to someone that has important/immediate editing to do and has not tested the workflow first. You are buying into something that has not been proven yet and will require some sacrifices until compatible software/hardware catches up.

 

[ajcourtney]

When I let the only 4GB clip in this project run to the next clip in the NLE, I don't see a stutter.

It's likely you wouldn't see a problem at the splice point. It's better to dump the audio into SoundForge and evaluate the wave file visually. When I did that with the Canon footage, I was able to count a specific number of lost samples (@ 48K/16-bit) that created a very specific audio "hiccup" at the joining point between multiple broken clips. Again, the rhythmic nature of rock music (in this case a live Collective Soul show) made every splice point totally obvious on the timeline in Vegas.

These were raw .mts files opened directly onto the Vegas timeline

 

[ajcourtney]

Not just to you... but to all that are waiting on Panasonic, sure it would be a nice gesture, but I wouldn't hold your breath. Why should they be required to fix a problem with someone else's software? This IS NLE specific! The camera follows the AVCHD spec, so it is up to the software to follow suit.

The file size restrictions imposed by the decision to use specific filesystems has nothing to do with the AVCHD committee. I contacted the consortium on this very point and did not receive confirmation that the AVCHD spec includes a requirement to (1) use filesystem "x" which would impose file size limits on video footage and (2) incorporate a software utility designed to sidestep this file size restriction and correctly reassemble camera clips that exceed any file size limit imposed by the particularly file system incorporated in a vendor's hardware product. Actually, they make it pretty clear that it is up to the particular vendor to decide how to logistically implement the AVCHD format, which, IMO, is one of the primary reasons we are left to deal with this problem.

Therefore, when you say it is the responsibility of the NLE vendors to follow suit (I assume you are implying that these NLE's are NOT AVCHD "spec-compliant"), your statement makes no sense because these NLE's are not "out of compliance." And that will be their reply - "tell Panasonic to fix its video footage".

So, the take home message here is there will continue to be a dance between the hardware vendors and the software NLE vendors where neither party will accept responsibility for this issue. And that is exactly how this problem has been playing out for the past 10-12 months.

Actually, as I said earlier, Sony does provide a transfer utility that correctly reassembles broken clips from their AVCHD consumer cams. I would imagine their footage would play back seamlessly in any NLE that supports native AVCHD footage. Unfortunately, I am not a Sony camcorder fan so that fact doesn't help me in any manner at this point.

 

[mcsmooth]

The file size restrictions imposed by the decision to use specific filesystems has nothing to do with the AVCHD committee. I contacted the consortium on this very point and did not receive confirmation that the AVCHD spec includes a requirement to (1) use filesystem "x" which would impose file size limits on video footage and (2) incorporate a software utility designed to sidestep this file size restriction and correctly reassemble camera clips that exceed any file size limit imposed by the particularly file system incorporated in a vendor's hardware product. Actually, they make it pretty clear that it is up to the particular vendor to decide how to logistically implement the AVCHD format, which, IMO, is one of the primary reasons we are left to deal with this problem.

Therefore, when you say it is the responsibility of the NLE vendors to follow suit (I assume you are implying that these NLE's are NOT AVCHD "spec-compliant"), your statement makes no sense because these NLE's are not "out of compliance." And that will be their reply - "tell Panasonic to fix its video footage".

The AVCHD spec may not REQUIRE a FAT32 filesystem, but did you ask if the spec is to recognize spanned files? Your logic does not make sense unless you can prove that the spec does not allow for stream files to be split physically and connected by the respective clip files. If that is the case (OR the camera does not do this properly), then you have a solid point. I just doubt it since some software was working before the camera came out and DVDs have been using a similar technique for many years which also use an mpeg2 stream and information files.

Your (2) would have no point if the files are indeed within spec. You wouldn't need a utility if compliant hardware/software is required to recognize spanned streams. People are asking for one because their software doesn't merge the streams properly, but the point is, it shouldn't be needed. I do agree it would be a useful tool though (even for people that don't have problems and simply want a single stream). There are a LOT of things I wish the AVCCAM viewer did, so I do hope for an update.

Keep in mind that the AVCHD "spec" is really just a modified blu-ray spec with some changed rules and restrictions. I do see your point about how there is some leeway on certain aspects that will give software developers pain anytime a manufacturer decides to do something different from what is already practiced. The major non-standards are in media, file structure and formats in the stream. The toughest aspect is the streams since software companies have been mainly testing with lower bandwidth 1080i files. File structure differences should be easy fixes whether they follow existing specs or not. What is being argued is if spanned stream and clip ability is assumed by all compliant players. I would think a good way to test this is to convert the file structure to blu-ray and load on a player that does not recognize avchd file structure. Those players are very strict to the standards, but will play back the streams when named properly.

The only way for Panasonic to "fix" their camera to the way people are requesting would force them to stop using FAT32. That would require a major firmware update if possible at all... I would expect that to never change on this model, it would piss off a lot of people. So unless they change or provide an optional second filesystem, files will never be under 4GB.

 

[ajcourtney]

No, I did not ask specifically about how the spec defines the proper method of handling spanned files. I don't think that thought process would have occured to me because all of us who were dealing with this issue were trying to eliminate the various clips by resplicing them back together to present a single file to the NLE. And no, I am not callilng for Panny to "fix" their cameras because I agree the hardware is functioning capably. Unfortunately, the incompatibility exists in a crucial area of software/hardware interfacing that enables each party to blame the other in such a way that nothing may ever get done to address this issue in a satisfactory manner for all users.

You mention that some software was working with spanned AVCHD files before the cam was released. That's news to me because everyone who tested this problem with Panasonic and Canon consumer cam AVCHD files experienced the same problem regardless of the NLE used. And converting out of AVCHD did not solve the problem either - I can personally speak for Cineform since I tried that format. Others tried whatever they had at their disposal - didn't matter.

I find it extremely interesting that Sony provides a utility to resplice these files back into the original contiguous clip to present to the NLE. Since they have taken this route and they run the show in Madison, Wis, I have serious doubts that Vegas will ever provide seamless editing of spanned files. And that's a real shame since I have been a Sonic Foundry user for almost 14 years and prefer that platform.

However, if this problem is resolved by converting out of AVCHD and into DVCPro HD, then I would have no problem doing that since I have always tried to convert out of AVCHD as soon as possible for editing purposes. Cineform provided that utility but was unable to resplice the clips, so the splice point problems were identical whether I was editing native AVCHD or Cineform. I doubt that DVCPro is as good of a codec as Cineform, but perhaps someone with some experience using both formats can chime in here. If it's good enough, then that's probably good enough for me.

 

[DaFireMedic]

Sorry to bring up the old thread, but I'm ready to buy (possibly) and need the info. I felt it better than starting a new thread.

I have $3500 in hand and as I mentioned am ready to buy the HMC-150, but I need to be able to shoot and edit seamless long form video. I read the entire thread but didn't really see a clear answer. I know that Barry said that the 150 is set up to record 12 hours of contiguous video, but I need to be able to get it into my NLE and edit it as such as well. I'm not concerned where the fault lies, only that there is an answer via hardware or software to get seamless transitions without losing frames between the 4gb clips and into my edited video.

Does the DVCPro conversion do this reliably so that I can have a reasonable degree of confidence in not losing any frames? Or is there another way?

If it helps I have Vegas Pro 8 and Premiere Pro 2.0. I am also willing to invest in CS4 if that would make the difference. Converting to Mac is not feasible for me right now.

Thanks in advance

 

[Averdahl]

Does the DVCPro conversion do this reliably so that I can have a reasonable degree of confidence in not losing any frames? Or is there another way?

Panasonics AVCHD to DVCPRO HD converter has passed all my tests i have done regarding this issue. PPro CS4 cannot yet "stitch" clips together properly. I hope an upgrade in the future takes care of it.

I do long shootings myself and i use the cameras HDMI to capture the footage with my Intensity Pro. Yes, the captures are realtime but i get seamless video so its worth it.

/Roger

 

[Barry_Green]

To reiterate a millionth time: the footage works. The camera works. The footage played back from the camera works. If there are issues, it's software issues and they'll need to be sorted out by the software manufacturers.

 

[DaFireMedic]

To reiterate a millionth time: the footage works. The camera works. The footage played back from the camera works. If there are issues, it's software issues and they'll need to be sorted out by the software manufacturers.

First of all Barry, thanks for replying.

I understand what you are saying and I read that before in your posts, but thats not what I was asking. I'm not complaining about the camera nor about Panasonic. Clearly the footage works in camera, which is fine if you are turning the camera over to your client and saying "here is your video", but most of us do not do that, we have to edit it and would like to keep our camera.

There is clearly an issue here (with the software manufacturers or otherwise). What I am asking is if there has been a software solution yet, some way of making sure that the clips can be made seamlessly contiguous in post, where it matters.

Averdahl: Thanks for the help, thats the kind of info I was looking for. This looks like a great camera and I am fine with converting to DVCPro (or using a BM Intensity card) as a workaround.

 

[floz23]

simple workaround

simple workaround

HOLY CR**!

I almost cr**ed by pants when I recorded my first shoot, spanning the 4gb fat32 limit. I put the two files into vegas and assumed they would span automatically, with no breaks... WRONG, either with 8.0c or 8.1. Thank the lord someone here suggested using the copy command to join the clips. The (windows) command line i used was

copy file0000.m2t /b + file0001.m2t /b + file00etc.m2t /b output.m2t

Whew!!! the resulting m2t, output.m2t, played back PERFECTLY in vegas. It seems that the 4gb limit will interrupt the GOP structure of the m2t, and the copy command simply joins the file, as if the gop were never interrupted in the first place.

anyway, i hope i save some other people on here from having a heart attack.

-Adam