Search for elusive "Bombproof Workflow" & p2 store & Genie

K

kstnate

Active member
Ok. I will try and keep this a reasoned discussion rather than a rant.
It seems to me EVERY workflow I have seen has an occasional "Issue."
Obviously, a lot depends on your edit system. I'd wager to say that the majority (but not all) of people who are using HVX200 are FCP users. I became one because of this camera. Also, thanks to p2 Log even more mac users are in the field pushing data around. I would be very curious to see what P2 users in PC world experiences have been, but more on that in a minute.

So, in a mac world, you have three popular choices:
1) firestore. I do not have direct experience with this, but I don't think I am fond of it. Wasn't the whole idea to get spinning fragile hard drives away from camera? plus corruptions due to bad cables, plus a another battery source, plus weight plus (I think) some limitations on record choices (please feel free to correct me on this) in a all purpose scenario I give it a thumbs down

2) P2 card then import to laptop. Pros: P2 card rock solid no moving parts or cables on camera. with 2 8 gig cards onboard record time is fine for me. disadvantage- I don't know about you but after i load FCP and a few other things on my laptop there just isn't enough room on the drive for footage of any consequence. This means an external drive, and this means an elaborate workstation. Fine for studio, BAD for just about everything else. Plus in this scenario you really need to have a data manager I think. I want the option of not having that- doing it myself. So, also thumbs down on this workflow in most cases, but see below.

one big issue in this system, a client used it in this method, used a laptop to copy to external drives, opened in FCP, everything looked great, then later copied from those drives to bigger drives. Those drives ended up having corruptions- about 20% of footage! Happy ending, corruption was not same on same folders so between 2 drives of info there was at least 1 uncorrupted copy of each file somewhere. Do not know if they were using p2 genie. Again, this is UNACCEPTABLE.

3) P2 card to P2 store, then later to hard drive. Pros: p2 card rock solid. Love the p2 store portability, lets me do 2 hour interview w/no breaks as long as I have my little box with me. wish it had more capacity, but what are you going to do? I bought two of these guys, so I guess you know what workflow I have adopted. So BIG thumbs up as far as this goes. but that is not as far as it goes. Like a highway that stops dead a mile before the destination, getting the stuff off the p2 store seems to be something panasonic just didn't think about! Sitting in a hotel room copying partition after partition, hovering over your laptop (did I just copy no name #4 or do I need to do that next....) one partition at a time. This is UNACCEPTABLE in a modern workflow. fine for personal projects or student films, but otherwise.....

P2 GENIE TO THE RESCUE, OR MAYBE NOT:
So along comes P2 genie. viola! a magic bullet, copy an entire p2 store in a click. Sure, backing up a full p2 store to two external drives (60gig x2) takes about 2 hours, but so does charging batteries. And it happens without me watching the damn thing. GREAT. I am in business. Only I am not, not so much. I have found p2 genie for mac in it's current configuration to be buggy. Some are known issues. Don't put strange characters into your filepath, or footage looks like it copies, but it doesn't. Don't try and copy footage to a drive that doesn't have the room for it, again it LOOKS like it copies, but it doesn't. Both VERY dangerous situations, but possibly fixed in next version (I hope) There are several other annoying bugs I won't get into here, except one.

But now I had an event that has made me act like Ronald Reagan. Trust, but verify. Since p2 genie doesn't verify, (unlike p2 store) things can happen. for me they just did. Copied stuff off p2 store (6 8 gig partitions) to two external drives. (redundancy) Should have taken about two hours. Took about 3 and a half. Hmmmm. Checked footage with P2 log as I am now a jumpy paranoid data freak. Actually it isn't paranoia if things are really out to get you. partition #3 everything copied EXCEPT THE ACTUAL VIDEO. Thumbnails, yes, but "Video" footage is empty. I check the second drive. there isn't even a "Video" folder! Gahhh! One drive is MAC formatted, and one was FAT32 (again in the interests of belt and suspenders) so it appears not to be formatting issue. Fortunately, I caught problem, and was able to bring the video in, problem solved. But I don't want to have to check my work like I have a green film loader who might have loaded things backwards.
Now I can't get too upset at the moment because p2 genie is new software trying to fill a need, and I am not sure this is a p2 genie problem, a mac problem or a p2 store problem. If you drive that last mile paved by locals instead of the Highway commission (read Panasonic) you take some flaws. At least for the moment. But this has to get fixed.

DISK IMAGES
I have done disk images, but in addition to possibly being voodoo, it puts me back to square one doing one partition at a time. Also I have had issues doing disk images from the p2 store. Sometimes the p2 store essentially crashes and dismounts and lots of red blinking LEDs are involved. So I don't like this either

QUESTIONS FOR OTHERS:
1) Has anyone had missing or corrupt files copying from p2 card directly to a drive (bypassing p2 store) using a MAC. Were you using p2 genie or not?
2) has anyone had missing or corrupt files copying from p2 card directly to drive using a PC? Were you using p2 genie for windows or copying?
3)has anyone had missing or corrupt files copying from p2 store either with or w/out p2 genie for windows in a PC environment?

All I WANT IS A BOMBPROOF WAY TO GET DATA TO A HARD DRIVE. AM I ASKING TOO MUCH?
p2 store seems rock solid getting data in. so maybe just export into FCP or avid using p2 store. But I have heard occasional issues with that as well. I don't want 8 different buggy ways to get it onto a hard drive I want one solid one!

I see at the head I was also going to talk about avid but this post is long enough for now. I will also be sending anders (author of p2 genie for Mac) a detailed bug report to aid in future versions, as despite its problems it seems the best hope, unless the lost data (like all those socks in the dryer) is an inherent problem in MAC or the P2 store or the p2 store WITH the mac.....
I am done for now. sorry for the long post.
 
Yes you are...

Yes you are...

All I WANT IS A BOMBPROOF WAY TO GET DATA TO A HARD DRIVE. AM I ASKING TOO MUCH?
Click to expand...

I'd have to say, yes you are. For several reasons really. First off, look at how new the technology and workflows are. Nothing new is rock solid out of the box. Next, if you're an avid reader of this forum, you'll see how many 'new scenarios' are constantly appearing in threads. New technology needs time for these scenarios to occur and get flushed out in the open. Only then can they be dealt with. In an industry that is so often all about thinking outside the box, creativity, and ingenuity, its hard for hardware and/or software developers to cover every if, and, or but out there.

Furthermore, you are technically dealing with devices that are known to fail. P2 cards are simply a type of hard drive (just specialized). Your computer (whether an Apple or an IBM/IBM clone) have tones of parts that can easily fail on you. Most relevant here being your own local hard drive. So even if we can get the material to the P2 cards in a rock solid manner, our hard drives (Raid 5's or not), are far from rock solid or 'bombproof.'

Frankly I think the best you can do is try to keep your cool, be patient with things, and have tones of redunant backups. Shooting to digital video cassettes wasn't made rock solid over night (and isn't necessarily even there to this day) and shooting to tapeless media won't be rock for a while to come (if ever).
 
Hi Nate,

I don't have the solution, but I bet it comes down to file/path naming convention issue. If you read my post and Filmstox's here: http://www.dvxuser.com/V6/showthread.php?t=66261, you can see we had sort of the reverse problem. Everything copied to our backup correctly, but then FCP said that there was an error corruption with the clips we were trying to import. This had to do to where the files were in the directory structure, not what medium they were stored to.

So like you said earlier, P2 card to P2 Store is rock solid. The engineers at Panny always know what structure will be coming from a P2 card, and have an established way of writing that data to the P2 store.

Now enter us mere humans (those that are lower than engineers) and our often failed drive path naming conventions. We might call a drive hanging 3 firewire connections away "2006-July-FCP" and then have an "active-July" folder containing other folders with client names, import folders, FCP Capture, etc. So while that structure makes sense to us, it somehow "whacks" things like P2 genie and P2 importer due to an improperly delineated path name, or excessively long path name issue.

So it may not be a solution, but maybe try a much shorter data path when using P2 Genie to see if a shorter name resolves the issue. It may be an issue of "it is just the way it works" until it works the way it should for us non-engineer types.

--

Good luck,
Rob
 
Nate,

Sorry, one more thought, and I think it backs up my file path hypotheses. After leaving this thread I went back to the workflow area and it was sitting there in front of me "New work flow from Apple" by Jeff Deveraux. The link is here: http://www.dvxuser.com/V6/showthread.php?t=57651, In essence this makes all the sense in the world from the engineers standpoint. The data always stays in its package, and when mounted, it is always at the root level. Since it is an error checked copy, everything is guaranteed as much as it ever can be in dataland.

So again, not a solution, but a least a better understanding as to why things may be running amuck.

Ohhh, and this is my 100th post. Do I get a ribbon or something? :-)

--
Rob
 
Maybe disk image is the way to go, but at the moment working with the p2 store (which Ireally would like to continue to do) seems to cause occasional "In out Error" and causes p2 store to dismount etc. plus, I have to make copies of each partition manually (since no one has made a disk image automatic copy program (yet). this meands I am sitting in front of my computer, hovering, waiting to click a new operation every 10 minutes for two hours. Me no like.
I realize this is a new workflow, but it seems like rather than time shaking things out (solutions for each path getting sorted out) it just seems to get buggier. 8 10 12 different ways to do something (fascinationg, and good) but problems with ALL of them. Ugh.

I know hard drives are not to be trusted, so I try and copy everything twice to two phisical drives (I have had my C drive fail a couple of times on my PC over the years so I am healthily suspicious of them, but feel pretty comportable with having two copies)

I will experiment with file structures, at the moment I suspect it is not it, but maybe. I had started a copy with a longer filename (in otherwisse the same folder) and that copied correctly. So shorter filename is where problem manifested itself. maybe I will try copy again in exacly same scenario see if it is duplicatable. I fear it may not be.

Also, when my client had those corruption issues, I experimented with some p2 data, moved it back and forth between two drives (one fat32 one mac format) about six times. Everything was fine.

I read both posts referenced. Thanks for linking them here for others.

Anybody out there dealt with 20+ hours of footage that you have not had a file/data problem with, and if so, please schare your stats (computer, copy method etc.)

Thanks.
 
Im very interested in getting a full report :-)

I'm working on MD5 verification right now, but do you think that creating Disk Images is the way to go ultimately?

You said that there are other issues besides the ones you mentioned, could you please send an email with them ;-)
 
just sent email with more detailed bug report. I have no idea whether disk image is the way to go. Like I said I have had issues with getting disk images to implement sometimes off of p2 store, but then again, it appears as if p2 store is using disk images of some sort to copy cards and that part of the workflow seems very robust. if sombody could figre out software wise what is happening when you have the little "verify" switch on on the back of p2 store I would love that again for the step of dumping off of p2 store.

Never really heard of disk images until a month ago, don't really know what MD5 does for you either. But I have a hard time believing computers are so messed up that one out of 25 file transfers is going to get messed up.
 
Nate, I'm with you. I just had exactly the same problem come up last week and entered a parallel thread about it today.
http://www.dvxuser.com/V6/showthread.php?t=66323

I often will be loading 12-15 4gb P2 cards onto my P2 Store, and I can't even begin to say how much I hate to sit by the computer tending one copy after another when I have just wrapped on a 15hour shooting day and am looking at another in 6 hours. So P2Genie seemed like a lifesaver...I love it. But then I just lost 2-cards worth of footage (first time in 6-months), and that is going to cost me a 1/2 day of reshoots. Yes, I know it's new technology, and yes I know I should have known better than to trust. But when I ran the program, I saw all the card images on both backup drives and did a spot check on the footage...unfortunately, not the cards that had no Video folder!

In this day and age, I -don't- think it is too much to ask for a bulletproof workflow for those of us working day in and out with the technology. Frankly, though I love P2-Genie (and it may be the fault of the lost data), I blame Panasonic for not thinking the workflow all the way through. Manually coping P2cards one at a time just is not practical. OK, so that may sound like whining...but hey, the place the HVX is making a HUGE impact is in low-budget independent film, and in that world schedules are super tight, and crews are shorthanded, and still we have to get the story told.
 
another factoid which may or may not hepl was i was doing interviews, so the clips were pretty long. Don't know if that had any relavance, but it sure made checking all my work easier (only 4 clips or so per card to check rather than say 20)
Also interesting we had exact same problem at similar time. Hmmm...
I do regular diagnostics of my p2 store via my PC, last check was 7/15/06 with 59 hours on that p2 store everything checked out ok. I will do it again, but only after the footage currently on it is safely imported into FCP in a few days. maybe that will shed some light.
 
OK, so here is yet another problem I have just run into with the P2 Store. Today, I was shooting pickups for a feature (one of which was for a shot that was part of the P2-to-P2Store-to-P2Genie-to-Mac missing VIDEO folder mystery), so I was ultra careful, and there were only 4 cards worth of shooting. At the end of the shoot, I connected the P2Store to my Mac Book Pro, loaded up P2Genie, and was quite surprised to see only 1 Card. So not panicking, I rebooted everything, and then the P2Store came up with 3-cards (not the 4 cards I knew I had stored on it). Nothing I did could make the elusive fourth card appear. So, back at home, I attached my P2 Store to my PC (not my mac), and guess what? The missing P2Card image is there and mounted -BUT- the three that the Mac mounted are not mounting on the PC!?!? Rebooting- same outcome.

Before the shoot, I had run the full diagnostics on the P2Store: zero defects, and then done a full reformat of the P2 Store using the Panasonic P2Store Manager programs's "All Erase" option. So it was as clean as I could make it before the shoot.

Has anybody seen this before? This is getting a bit silly.
 
OK, so here is yet another problem I have just run into with the P2 Store. Today, I was shooting pickups for a feature (one of which was for a shot that was part of the P2-to-P2Store-to-P2Genie-to-Mac missing VIDEO folder mystery), so I was ultra careful, and there were only 4 cards worth of shooting. At the end of the shoot, I connected the P2Store to my Mac Book Pro, loaded up P2Genie, and was quite surprised to see only 1 Card. So not panicking, I rebooted everything, and then the P2Store came up with 3-cards (not the 4 cards I knew I had stored on it). Nothing I did could make the elusive fourth card appear. So, back at home, I attached my P2 Store to my PC (not my mac), and guess what? The missing P2Card image is there and mounted -BUT- the three that the Mac mounted are not mounting on the PC!?!? Rebooting- same outcome.

Before the shoot, I had run the full diagnostics on the P2Store: zero defects, and then done a full reformat of the P2 Store using the Panasonic P2Store Manager programs's "All Erase" option. So it was as clean as I could make it before the shoot.

Has anybody seen this before? This is getting a bit silly.
 
Paul,

That is an interesting post. For the first time recently I used 10 or 11 of my P2 Store partitions during a shoot. When I went to copy those files with P2Genie it went into "take forever to copy" mode for the first time in spite of very efficient workflow here to for with the came set up. This is to a MBP too. I canceled the copy process and immediately lost 3 or 4 partitions. I shut the P2 Store off and restarted it. Now I only had 1 partition on my desktop and I got a solid red#2 error light (which I believe says in the manual send the unit into Panasonic) Anyhow I shut everything down and didn't mess with any of it for a few days due to other things. When I came back to it all I started the P2 Store up and lo and behold there were all my partitions and P2Genie copied them all over no problemo. I don't have a clue but my suspicion is possibly the USB2 implementation in the MacBookPro or possibly heat related read errors in the P2 Store - but that's total conjecture.
 
All the problems reported with the P2 workflow on this forum have been computer software problems, computer hardware problems, user error, disk crashes, etc. Under operator error, you find users not knowing they have to set up a P2 store for Mac vs. PC, not knowing there are certain characters they are not allowed to use in a path name, etc., etc. Sure, there has been one case of an actual defective camera, a couple of defective cards, but 99% of the case have had nothing to do with the video portion of your program. It's computer stuff.
In the tape workflow, you had moving parts that broke. And how many times have you or your editor had to "clean the heads" on your P2 card? As the guy says in the joke, "Parts is parts." Workflows is workflows. We use cards, P2 Stores, FS-100s, and the only problems we've had are one crashed disk (brand new LaCie) and one dumb effort to copy a card using a daisychain of P2-to-iBook-to-Bus_Powered_Drive-to-FireWire drive.
And the "disk image" method is pure voodoo. Useless. There is no advantage at the hardware level or the software level to store your clips using this method. That's not what a disk image is for. Bits is bits. A computer writing a disk image to a hard drive can make the same errors if it's copying any file to a disk drive. A disk image IS a file, for crying out loud. A file.
 
so what if it is software or hardware problems? As they say problems is problems. Yes, I agree the p2 card is very robust. More robust than tape. But at current cost part of the workflow currently DEMANDS some sort of field offloading which involves P2 store or a laptop usually. So a problem in the workflow is a genuine problem. Yes, admittedly there are lots of "d'oh" moments as we all learn the workflow (and learn more than we ever cared to know about bus speeds, etc) But disappearing footage is a very bad thing. very very bad. I am not trying to badmouth HVX200, I own one and badmouthing it is pretty much like shooting myself in the foot. But if a client wanting to rent it asks if there have been problems I find myself on thin ice, trying to very clear and hope the producers don't just hear "Blah blah blah, yes." I want this camera to succeed. I think most of us here do. It makes great pictures. I knew buying it meant some additional time dealing with media I don't do with regular cameras. I didn't think it meant transferring each partition one at a time after the job (w/out p2 genie could be 2+ hours in front of computer) then checking my work with some sort of viewing program because the bits may for some unexplainable reason have disappeared into the either.

I have been fortunate in that there have been enough problems from other users that I was paranoid and checked my work, and ultimately have not lost any footage.

Anyway, I am currently importing footage into FCP. Once done I will run diagnostic check on p2 store and report back. Currently my (totally unsubstantiated) theory is problem does not lie in p2 genie (but it would be really nice if a verify check were done so if a problem arises I know about it, something Anders I am sure is writing code for) but something to do with Mac and USB on P2 store. I base this only on the feeling that footage dump into p2 store seems very robust, just getting it out via the usb port into the computer seems arbitrary. Sometimes copies go fast, sometimes, for no apparent reason they go really slowly. And sometimes (in my case once in about 50 hours) something goes missing. MAc seems more firewire based and PC more USB and we all know which computers Panasonic has in their factories. PC.

Anyone out there experiencing a data problem working only in PC world?

BTW I have inadvertantly hooked up p2 store to PC while in MAC format. It doesn't completely not work, which could confuse people, since it works just enough to be wierd. Those of you hopping between platforms keep that in mind.
 
Oh, yes, I actually hope the disk image thing is crap, I prefer the p2 genie method. at least when everything is working like it should. But I will take whatever measures necessary to have a bulletproof workflow as long as it doesn't involve hand writing out the bits into notebooks (And you tought usb was slow!)
 
Thanks for that reply Nate, you saved me a lot of typing! I agree completely. I suspect that most people, not all mind you, that advise "chillout, its only new technology and someday soon the kinks will eventually work out", have in fact not experienced the joy of connecting up the P2Store and only seeing part of what the day's shooting entailed...with the producer hovering over your back.

I don't buy the just take it easy approach at all. These are professional tools...perhaps at the lower end of the spectrum, but pro-tools all the same. They are cheaper because the camera does not perform all the same tricks that a Varicam does. They are not cheaper because Panasonic asked us all to agree that it would be OK to loose footage intermittently while they figured out how to make it work - not perfectly, but fault tolerantly. I purchased the P2Store (at what I thought was a fairly ridiculous price) because I wanted to adopt the mainline workflow that Panasonic had developed for this camera.

How much footage is it OK for me to loose over the rest of the long years of DP work I have ahead? ZERO

OK, I'm done ranting. I do agree that all roads seem to be leading to the USB implementation. On my MacBookPro, workfellow, the P2Store comes up with only a fraction of the partitions loading. After rebooting both, they all come up. Then there was the strange anomaly I just wrote about a few entries above.
 
status report

status report

I ran diagnostic on my P2 store- came back "Status OK" (Average 9.5 Max:122msec, min:8msec, Error count:0) I did the test before I erased data. Test took about 2 1/2 hours. I am now doing full erase (not Quick format) using p2 store mananger for PC.

So in summary, my data problems:
initially went to copy a partially full p2 store to a drive during shoot day (had some spare time) I changed project name in p2 genie but failed to close the side window that lets you into that menu before copying- so project was saved under previous job name. I tried to abort, but couldn't so I just let it cook.

That night I tried to more formally copy the p2 store using p2 genie backing it up to primary mac formatted external drive and 2nd copy (via p2 genie) to a physically identical drive but formatted fat32. Copy time was a lot longer than normal. Should have taken 2 hours, it took nearly double.
Checked my work via p2 log- found copy of partition #3 did not have video, on either drive. If I had erased p2 store before catching this it would have been bad.
Tried doing disk images but had "in out errors" several times and aborted attempts to do disk images. but sometimes it worked. Unclear if the aborted images were solely on the #3 partition.
Checked first copy (see first paragraph) and video footage HAD made it off the p2 store at that point. So I just copied it off tat folder to where it was missing and I was good to go.
Tested the possibility of filepath problems by doing a p2 genie copy off the p2 store with the same data (this time only making one copy) and keeping the same filepath except I spelled "interviews" (Part of filepath name) backwards so it had same character lenth and same characters, just in different order. Copy time was long (too long in my opinion) but footage all made it over.

So it could be
1)an untracable gremlin
2) a problem with USB and mac and big file sizes that causes mysterious data loss
3) possibly something to do with doing two copies simultaniously in p2 genie, mixed with something above. (don't think it is that, but I just finished wiping that p2 store so can't test that hypothisis exactly, but my other p2 store copied 2 copies over in a timely fashoin via p2 genie no problem.

Boy I wish I knew what happened.
 
I just had the problem where my computer only recognized only a handful of partitions on my P2 Store. It came up as 6 partitions, and I knew from working with it the night before that I had 7 partitions. I ejected the partitions off the mac and connected it up again. This time 4 partitions showed up. I then powered off the P2 Store and powered it on again and all 7 showed up. Why does this happen?

I need to really mark on the P2 Store somewhere how many card I downloaded to it, because if I hadn't known that there were 7 partitions from using it the night before. I probably would have just copied over the 6 and wiped the P2 Store clean basically deleting a whole card worth of clips. Has anyone figured out why this is happening to people and how we can avoid it?
 
don't know. I haven't had that happen yet. What is your order of connecting? power up computer, connect cables, power up p2 store? or some other order? Maybe that has something to do with it?
 
You must log in or register to reply here.
Back
Top