Workflow for classical music recording session / edit, on a tightish budget

[ullanta]

Greetings! I'm looking for advice on a pretty specific situation that rarely if ever seems to be addressed. Perhaps it's too complicated (which would explain why classical music performances are almost always horribly (mis)synced in films).

These days I am occasionally called upon, during recording sessions for classical music pieces, to also record video. In general, this involves ensembles and halls that require very tight and short schedules, and long pieces, with few complete runthroughs and many short takes aimed at getting a specific section or measure of the music correct. Video is not the primary concern, and the usual simple process is to just run one wide camera throughtout the recording session, and have one close up camera that is moved between takes in an attempt to get coverage of all the performers during repeated takes of sections of the music. There's generally no timecode equipment.

The final edited audio is the main product, and generally has a pretty high edit density, at least in many parts. Many hundreds of edits for each project, often fractions of a second apart.

So, after the audio edit is done, the desire is to construct something from the video which looks like a continuous performance, with many closeups of musicians, that syncs to the edited audio. When I say syncs, I mean that my clientele are disturbed when vibrato is not synced perfectly, when fast piano passages are not in perfect sync, when bowing or fingering are slightly off, etc.

The most basic tricks - shots that don't show hands, etc. - are often considered unacceptable, especially in the most difficult passages of music.

What I'm wondering is if anyone has any experience with something comparable, and/or any ideas on workflow for a situation like this. I've tried many things, with varying degrees of success, but none have proved to be anything other than excruciating. Even assembling a "baseline edit" where the video clips associated with each little audio bit in the audio edit are assembled to match is a grueling business. Without timecode (or with it!), is there a way to take a ProTools edit and automatically have the video clips assemble themselves to match?

And then, less mechanistically, can anyone recommend workflows for auditioning candidate close-up video from different takes of the same material?

Finally, any ideas on which editors might be best (or worst) suited to this kind of thing?

Cheers,
Barry

 

[ggrantly]

I know you are looking for advice, but I will give you the obvious. If they expect good video, it will take multiple cams, lighting, and a genuine effort if they have any serious expectations. Makes me think how often videographers get roughed up on the audio board with a question like "I want great audio on the cheap, what should I do?" Sorry, I'm not trying to dump on you, but your customer needs realistic expectations relative to the dollars and effort expended.

Grant

 

[ullanta]

Hey, like I said, video is not the primary goal; and furthermore, having all that stuff (and personnel) in the hall is not desirable on an audio recording session. I understand your obvious answer, and hope i avoided looking like I was aiming for top-quality video in this circumstance. Certainly there's no financial benefit in this to my clients - primarily they're composers who want to entice people to perform their pieces. I'm just looking for advice or commentary on this as an interesting problem. However, even with a larger budget, what would people do? As i said, even in major films the way image relates to sound in classical performances is usually laughably bad.

Meaning, even with all the skilled crew and equipment one could dream of, how would one go about assembling the many takes of video into what looks like an actual performance of the edited audio?

Now, from all I've read, the common technique used for non-classical music video, and likely for some classical footage used in films, is to have the performers play along to a recording (i.e., fake it) with many cameras and many takes, with the idea that since they're faking along to the same recording that will be used in the final product sync will be OK. But it's not! It don't look real. And in my case, it's not going to happen. In addition to budget and scheduling constraints, the nature of long complex classical pieces is that they never come out exactly the same twice... even faked.

There's no real answer. If it does work, it is a lot of work and some luck. But I'm looking for any ideas people may throw at me.

 

[paulears]

Having spent a fair time recently recording a classical audio CD, I understand 100% the problems you are having. My attempt to do something very similar - for experience for me, not for the client, also had exactly the same sync problems. For the benefit of others who have not done these recordings, it's like doing a multicam edit but 100 times worse! You record perhaps ten versions of the piece. Sometimes you actually get to the end with mistakes dotted around the musicians, or sometimes they just stop, so in the end, you have to try to match perhaps half a bar of music from take 6 to a section from take 2 which is joined to take 12 - hence the huge number of audio edits. Sometimes, it can be just a single note! The more musicians, the more chances of mistakes, but worse, less chance of clean edit points. What I might try to do for the next project - BEFORE the audio edit, is to not edit with the audio editor of choice, but to try and do it in the video editor so that the video tracks get cut and moved around at the same time, but with their tracks turned off. I'm not convinced PP, which I use will be nice to do this with, as audio tools are more limited - BUT - if the audio edit can be done, then I have the right video tracks stacked above that I can then edit vertically between cameras, but not ever move in time, unless they are generic pretty shots I can use to cover gaps where no video track looks good.

It will be a horrible job. I also discovered that classically trained musicians cannot mime - simply no idea how to pretend to play what they actually did - all they do is play the same dots again, but differently - aaaagh.

Probably got one of these happening in february - so I'll perhaps risk it?

 

[ullanta]

Probably got one of these happening in february - so I'll perhaps risk it?

Sure, it's fun if there's no pressure! I have done a number of these, so have some experience of what doesn't work... just haven't yet found what's best. My main advice would be to make every effort to have cameras covering everyone at all times! Then you can hopefully substantially reduce the need to use video takes that are NOT related to audio that's used in the actual edit.

Also, in the past, I've had some people willing to say that the best run-through would suffice for the video, and some people willing to do as you've suggested and make a "for video" edit that's less perfectionist, and ideally which could be done in the video editor. But especially composers trying to get definitive first performances of their work want that final edit! And it's a fun challenge.

 

[paulears]

Best ones I have done were where the composer was present, and we did the recordings in sections that would edit perfectly. So instead of doing long repeats and the danger of somebody else messing up, we only needed to go perhaps 24 or 48 bars then a stop. The musicians thought it odd at first, but it worked so much better - the worst case was 5 takes for a section. It also requires very detailed recording notes that make sense a week or so later.

 

[ggrantly]

Ok, now ill give you an idea. If you could have three cams as a minimum, one of which is on a crane, and the crane is on a set of dolly tracks in front of the orchestra. It would be best to synch all this with time code which will require some decent cams which can be jammed. Do your close up performer shots with the actual recording in one take. All your closeup will be in perfect synch. Then add in additional footage which isn't so synch critical which could be done in additional takes. Although this would require more effort up front, it would require much less editing and would likely require much less musician time and studio time. A good storyboard for your shots would be needed. For example, know what your two mobile cams are going to do while your crane is moving down the tracks for its next shot. One of the keys is to keep your cams moving around with a fair amount of cutaways so it doesn't become a bore.

 

[Richard Crowley]

Customers (and audience in general) have become "veolcitized" to high produciton values by watching public television specials with high budgets and shooting schedules which accomodate high-quality finished edit sequences; or literally DOZENS of cameras hidden among the players for coverage of live performances.

Small "handycam" digital HD camcorders produce remarkably good pictures when lit properly. And there is greatly increasing use (and audience acceptance) of "sports POV" type cameras (GoPro Hero, et.al.) I have acquired some handy little adapters that thread onto a mic stand (or bracket adapter) and have a micro ball-head mount for a small camera. You can deploy them like microphones on stands and get some great "insert" shots of the action. I saw a piece on Youtube (shared on Facebook) of a child prodigy pianist playing at a large concert. It included some great closeups of the keyboard, and a wide shot revealed that the "keyboard cam" was a little camcorder shooting "wild" (not synced and not live-switched).

http://www.youtube.com/watch?feature=player_embedded&v=aaakFH8JIYU#!

http://www.amazon.com/gp/product/B001GWCC4I/ref=oh_details_o05_s00_i00

 

[ullanta]

Ok, now ill give you an idea. If you could have three cams as a minimum, one of which is on a crane, and the crane is on a set of dolly tracks in front of the orchestra. It would be best to synch all this with time code which will require some decent cams which can be jammed. Do your close up performer shots with the actual recording in one take. All your closeup will be in perfect synch. Then add in additional footage which isn't so synch critical which could be done in additional takes. Although this would require more effort up front, it would require much less editing and would likely require much less musician time and studio time. A good storyboard for your shots would be needed. For example, know what your two mobile cams are going to do while your crane is moving down the tracks for its next shot. One of the keys is to keep your cams moving around with a fair amount of cutaways so it doesn't become a bore.

I'm not sure what you mean - the "actual recording" is a complex audio edit that's compiled from dozens or hundreds of takes. And crane? dolly? in an audio recording session? Not gonna happen! ;-)

 

[ullanta]

Small "handycam" digital HD camcorders produce remarkably good pictures when lit properly. And there is greatly increasing use (and audience acceptance) of "sports POV" type cameras (GoPro Hero, et.al.) I have acquired some handy little adapters that thread onto a mic stand (or bracket adapter) and have a micro ball-head mount for a small camera. You can deploy them like microphones on stands and get some great "insert" shots of the action. I saw a piece on Youtube (shared on Facebook) of a child prodigy pianist playing at a large concert. It included some great closeups of the keyboard, and a wide shot revealed that the "keyboard cam" was a little camcorder shooting "wild" (not synced and not live-switched).

Right... I do this a lot for live performances; and for that, the sync issues are pretty trivial since there's just one "performance". In addition, you say "when lit properly", and I will mention for the sake of others thinking of attempting such things that this is usually both unmodifiable AND a given - that is, performance stage lighting is usually very good for these cameras to work with, and usually out of the videographer's control (at live performances).

But Richard, have you any experience with the situation I described above, of trying to craft a video performance to match the final edited audio of a complex recording session?

 

[ullanta]

Best ones I have done were where the composer was present, and we did the recordings in sections that would edit perfectly. So instead of doing long repeats and the danger of somebody else messing up, we only needed to go perhaps 24 or 48 bars then a stop. The musicians thought it odd at first, but it worked so much better - the worst case was 5 takes for a section. It also requires very detailed recording notes that make sense a week or so later.

Yes, luckily (or not?) many of the composers i work with are very good at the record keeping and focusing on the sections that are "imperfect"... but also tend to be into very complex timing. Sometimes we're just going repeatedly after one measure (recording one or two before and after for "handles")... sometimes 15 or 20 times (of course with long discussions between each at times!)... but as a warning again for others, this really needs a composer and/or producer who knows exactly what s/he is doing, 'cause it's very easy to find later that you haven't sufficiently covered (or have an audio problem or performance problem) one little section that ties together some other sections... and then the whole project is blown. The recording notes are essential for this, during the session, as well... but in the heat of the moment, under time pressure, and dealing with performer endurance and temperament (not to mention instrument tuning, changes of which can prevent certain edits)... it gets tricky. Also, there's something to be said for the emotional flow of a piece played through as opposed to in sections... so i at least always try to insist on at least a runthrough before all the detail work, and a runthrough after... and when things go well, the "after" runthrough is usually the bulk of the final recording.

I find that, on average, for a 10-minute movement I have about 50 takes of different sizes - and usually at least 3 or 4 of these takes are "just leave everything running and let us keep starting and stopping until we're satisifed with this little chunk"

 

[Alan H. Chang]

Without timecode (or with it!), is there a way to take a ProTools edit and automatically have the video clips assemble themselves to match?

And then, less mechanistically, can anyone recommend workflows for auditioning candidate close-up video from different takes of the same material?

Finally, any ideas on which editors might be best (or worst) suited to this kind of thing?

To your first question re PT session and auto video clip syncro, no. What you can do is export the recording session as a finalized medium, and import that into PluralEyes. The most current is v3, and does both video and audio syncro amazingly well. After all the clips have been synced, you can import that working session or environment from PluralEyes into FCP, Premier, or Avid.

To your second question, the more available option is to have each clip on their own channel strip with each clip dedicated to its own video output, ie. preview 1 monitor, preview 2 monitor, program monitor, and etc.

To your last question, and surely not that helpful, is to use what's best suited to your editing workflow.

Situations like these definitely requires tri-level sync where we have timecode, word clock, as well as video for post prod editorial purposes.

As an investment if you're ever going LIVE with a real milticam shoot, a Tricaster would do wonders. Wirecast, Anycast, BoinxTV, and all other variants.

 

[paulears]

Quick question. Will pluraleyes sync the video to the audio, rather than the usual audio to video?

None of the video editors I'm aware of can read an edited audio track, determine which take it was and then import the 'correct' video to match that section. That would be a great feature. We've got a system that lets audio be 'glued' to video, but the audio editors don't do the same things - most allow you to import video as a track, but not attach specific images to the audio line?

 

[Alan H. Chang]

Quick question. Will pluraleyes sync the video to the audio, rather than the usual audio to video?

None of the video editors I'm aware of can read an edited audio track, determine which take it was and then import the 'correct' video to match that section. That would be a great feature. We've got a system that lets audio be 'glued' to video, but the audio editors don't do the same things - most allow you to import video as a track, but not attach specific images to the audio line?

Yes! I also wish more NLE would do the same, on the wishlist.

 

[Richard Crowley]

But Richard, have you any experience with the situation I described above, of trying to craft a video performance to match the final edited audio of a complex recording session?

I think you made a direct hit on the nail when you said: "Perhaps it's too complicated (which would explain why classical music performances are almost always horribly (mis)synced in films)." It is an impossible gap between the shooting constraints and the performer expectations. There is no "tight budget" solution to that gap IMHO. Certainly if you had a timecode-recording camera for every possible camera angle, and a meticulous record of which takes over which bars at what (synchronized) time, and many hours in editing, you could produce an exact and perfect record that the musicians are seeking. But that requires equipment/personnel/budget well beyond what is available for typical situations, at least IME.

Which is why we are seeing "rock-music style" music videos which consist of shooting in various locations and "lip-syncing" to the edited studio sound track. For example the wildly popular (millions of hits) "Piano Guys" et.al. http://www.youtube.com/user/ThePianoGuys

 

[paulears]

I suspect Alan and I probably work on similar projects - and what we do is VERY different in terms of process from the usual music & video products. So few of the techniques are the same - and our editing is rather like where you have shot a dance video with one camera on three nights and are attempting to match shots - yet another very unpleasant and time consuming project style.

 

[Richard Crowley]

The expectations of having a perfect note-for-note, stroke-for-stroke, step-for-step visual representation of a composite assemblage (audio and/or video) is simply unrealistic. No video/cine production is made that way. The musicians simply don't understand how things are done and are fooled by big-budget Hollywood productions into thinking their requirements are realistic. That is what I meant by "velocitized".

 

[paulears]

I take your point Richard, but if a musician hears themselves playing, is seeing their fingers match what they hear unrealistic? Lip-sync errors are always considered dreadful, so from the musicians perspective, should their fingers not actually sync what we see? I fully appreciate the difficulty, but in an ideal world, they really should be fixable - it's simply that doing it is so tricky. You could find the video shot at that exact time, and hope that the shot was of what you wanted? My experience is it rarely is. Good audio and good visuals of the thing you hear being almost luck! Maybe it's just that replacing voices doesn't have to be as exact as a guitarist hearing a G chord and seeing an F - which happens so often in pop videos, let alone finger picking!

 

[Noiz2]

Alan beat me to it but Pluraleyes was going to be my suggestion. It was designed to sync multiple cameras based on their audio tracks. So if you make your cut audio the master then you should with some effort get the clips to sync to it.

The catch is that with so many edits it could get dicey. But if you just use it on the critical clips where you have a close up then it might not be so bad.

Working "backward" you will always have the problem of NLE's only working at the frame level, but you should be able to get better than 1/2 frame sync. The only way to get dead on sync is to cut the sound to match the picture not visa versa. I know that isn't an option but you might point that out to them - it's not a skill issue it's a media issue, you can't cut picture any tighter than a full frame so you can get a half frame from sync but any closer is a crap shoot.

 

[ullanta]

The expectations of having a perfect note-for-note, stroke-for-stroke, step-for-step visual representation of a composite assemblage (audio and/or video) is simply unrealistic. No video/cine production is made that way. The musicians simply don't understand how things are done and are fooled by big-budget Hollywood productions into thinking their requirements are realistic. That is what I meant by "velocitized".

Well, I should note that the musicians' expectations and understanding are not so unreasonable, and they certainly haven't been fooled by Hollywood productions - they hate 'em and are hoping to do a lot BETTER. But we all know it's hope. When I say they're picky and such, I don't mean that they are demanding the perfect assemblage - I mean that they don't want to put out any video that doesn't look like they're actually playing. And as a result the video is sometimes not put out into the world, without any hard feelings. But we all want to try to achieve something like this, and sometimes do.