PDA

View Full Version : Why is audio scrubbing wrong?



hdace
08-22-2015, 10:32 AM
There are dozens of bugs and improvements I would list if I had a couple of hours to spare. Luckily many other people actually do spend that time and LightWave gets improved. Thanks to you heroes! This is one "bug" that I have to mention because it's so darn important to get it right and I can't believe the LightWave engineers have got it wrong. In fact, I could be mistaken, but I don't think it's always been wrong. I think it's new or I would have noticed it before.

Lip sync is incredibly important. Duh! That means our basic tools for working on lip sync have to be good or it'll drive users nuts. Whether you're using morphs or mdd's from Face Robot, there's a good chance that when you finesse your characters' lip movements you're doing it in the Graph editor. You're creating keyframes on a curve to pinpoint the exact moment of a hard consonant like a B, P, M, or even an F or V. You're also studying the waveforms and listening to the audio scrubbing to determine the exact temporal location of those consonants. And then you make sure the keyframes and the waveform locations line up. With me so far?

LightWave's audio scrubbing is not following industry standard. Every other program I use will scrub the frame that I've just landed on. Lightwave scrubs the frame I just left! If I'm on frame 10 and I hit the right arrow button once, then I go to frame 11. I should hear the audio from frame 11. But I hear the audio from frame 10 instead. How does that make sense?

If my character is performing a "long" M (maybe for emphasis) I need to know the exact frame the M ends and the vowel begins. I can clearly see it in the waveform. I've got a keyframe on the last frame of the M and on the next frame the character's mouth opens. Whilst scrubbing, when the mouth opens I should hear the explosive beginning of the vowel. Instead, I hear the last frame of the M sound. I constantly have to remind myself that LightWave is scrubbing the wrong frame!

sudac20
08-22-2015, 10:51 AM
Agree Lip sync with Lightwave is a bit of nightmare

MonroePoteet
08-24-2015, 09:19 AM
There's an Audio Start Time option in the Scene Editor => Audio menu. Perhaps you could set it to -1 frame to align it they way you need? I don't know, never tried lip synch with LW.

mTp

hdace
08-24-2015, 01:36 PM
The audio's already lined up correctly. NewTek needs to fix this.

Dexter2999
08-24-2015, 10:18 PM
While I am all for better audio scrubbing, I have a little bit of a hard time believing you NEED absolute frame accuracy. I have been doing video stuff for over 20 years and three frames is obvious. Two frames off might seem "off" to some people while others won't notice at all. I have never met anyone who can spot a one frame sync error.

I'm talking one thirtieth of a second. To put some animation perspective on this, for decades cartoons were 24 frames per second and shot "on two's", meaning each drawing was photographed twice. For a total of 12 drawings per second. What I'm getting at is that the general viewer will be more forgiving your personal exacting standards might be.

Also, a REAL dope sheet can be quite usefull. Time it out in ProTools or Audacity. Tim Albee also made a super cool frame conversion app, the itty bitty something or other.

This not intended to be argumentative, but helpful in offering the suggestion of traditional work procedures and a word of consolation about your aggravation.

I have done projects myself that required frame accuracy for lighting and fireworks cues timed to SMPTE in the audio track. So I understand sometimes you do NEED frame accuracy, but sometimes you don't.

jwiede
08-24-2015, 10:29 PM
While I am all for better audio scrubbing, I have a little bit of a hard time believing you NEED absolute frame accuracy. I have been doing video stuff for over 20 years and three frames is obvious. Two frames off might seem "off" to some people while others won't notice at all. I have never met anyone who can spot a one frame sync error.

Seems like you're presuming the frame rate he's setting in LW is 30fps. If he's instead using the animation frame rate (24/12fps) then a frame-level accuracy requirement is reasonable.

hdace
08-24-2015, 11:05 PM
Well Dexter, I don't think about the good old days when I was an assistant film editor in London very often, but your post sure brought it all rushing back! My first job was syncing up rushes for BBC documentaries in 1983. If anyone ever screwed up the clapper board then sometimes the only way to sync up the rushes was to identify a unique phrase in the audio and then figure out who and where in the footage the words came from. I was always looking out for Bs, Ps, & Ms. So you can imagine I got a little obsessive about it.

A few years later I got a job working in Rod Howick's cutting rooms in Dean Street. He was the veteran animation film editor that worked on hundreds of Richard Williams's films and TV commercials. He foleyed and track layed the great sound effects for the Pink Panther at the beginning of the The Return of the Pink Panther in the early 70s. He did all the original editing (including sound work) on that work print of The Thief and the Cobbler that Williams hawked around for years before finally getting the cash to produce the real film. I learned a lot from him, and from all of the amazing animators that came and went. I produced hundreds of lip sync breakdowns for them over the years, both of dialog and music.

So then eventually I became an animator. And yes, I admit I'm a little obsessive about lip sync accuracy. The irony is that often those animators would come into Rod's cutting rooms with their hot rushes in hand and I would sync them up with the dialog on the Steenbeck, and it wouldn't look in sync!!! I went to so much trouble over the breakdowns and the animators went to so much trouble to transfer my weird version of phonetics onto their dope sheets and everything was absolutely accurate and still it didn't look right. So I would just retard the audio by one or two frames and suddenly it looked terrific. I learned that trick from Rod: when in doubt, delay the audio.

When in the dubbing theatre (Mag Masters) just a few doors down the street, Rod could always tell if the mag film had slipped a bit after being synced. Those old machines slipped sometimes. He could always tell to the exact frame how far out it was. Our regular dubbing mixer was in awe of him. He'd just ask how far out, Rod would tell him, he'd make the adjustment, and immediately it would be in perfect sync. Of course, we were working in 24/25 fps, so yes, it's a little easier than 30.

I admit that even Rod possibly couldn't spot a film one frame out of sync at 30. I sure enjoyed the flashbacks you inspired, Dexter.

But even you say that you're in favor of better scrubbing, and there's no reason why we shouldn't get it since every other editing and animation package does it correctly. Why can't LightWave? When you move one frame forward, and the mouth opens, you shouldn't hear the sound of the M or B continuing. You should hear the little puff of air that comes out when the lips part. Am I right? It's not a question of making sure the audio is lined up correctly because when I see it in the Graph Editor I can see plainly that it is lined up correctly and LightWave is simply playing the wrong frame of audio! I admit I'm a little pedantic, but isn't my sanity worth something to NewTek? After all, I pay them about 500 bucks a year for this software. The least they could do is care a little bit about my sanity! OKAY??!!!!

Dexter2999
08-24-2015, 11:26 PM
I envy your experience, sir.

What type of audio are you using (mp3, mp4, WAV)?
My best results have been from uncompressed file formats like WAV. Even then it still, depends on the complexity of the animation. Complex bone rigs always gak my system and frame accurate scrubbing is out of the question for my middle of the road system.

hdace
08-24-2015, 11:39 PM
I'm using WAVs. I always break out the head on its own to do lip sync work, whether with morphs or mdds. If it's morphs I save the envelope files for each morph and import them back into the main scene. If it's mdd's from Face Robot then the actor can never get it frame perfect so it's a question of editing the mdd curve in the Graph Editor. Then again save the envelope file and import it back into the main scene.

By using the breakout method you get really clean playback, often at full or near-full speed. It should all be very efficient, and very nearly is except for the highly irritating problem of constantly hearing the wrong frame of audio whilst scrubbing one frame at a time.

hdace
08-25-2015, 11:31 AM
I have now discovered that this is actually an intermittent problem. I've gone through some of my previous project files and discovered that sometimes this happens and sometimes it doesn't. Going to try to find any difference between the two kinds of scenes.

hdace
08-25-2015, 11:42 AM
Okay, it didn't take long to figure it out. Wish I'd found this earlier. In the Scene Editor, at the bottom, in the Audio menu, is the Audio Start Time... item. This is obviously where one sets the offset for the audio so it matches the animation. If one sets a positive number then audio scrubbing works correctly. If one sets a negative offset (which I require approximately half of the time) then audio scrubbing does not work correctly. Now I'll create a test scene and upload it to NewTek in a Bug report.

ianr
08-26-2015, 07:51 AM
You send it in CAPITALS , cos being grossly wrongly for many years,
really strange since NewTek what with Toaster product know how to
deal with Audio?BROKEN for HOW LONG.... the answer is embarassing.
Personally Audio Scrubbing with third party plugins was my answer
One was Joe Alters Lovely Shave & Haircut, now not avail he is solid
Maya dev now
But TAFA may help its only $200 now or d/l the demo.
Timothy Albees Facial Animation. it has also the best joysticker around
should you that on Chars faces.

hdace
08-26-2015, 08:34 AM
Thanks, but I like using RHiggit! for character animation (including face). I'm sure Timothy's solution is great, but RH's solution is just fine with me.

I got an email back from LW3DG and they said they were reporting the bug. So all's well, soon, hopefully...