View Full Version : Using 3D graphics in iPhone application - lipsynch?

Tom Wood
07-02-2009, 09:22 AM

I have some 3D characters created in LightWave that I'd like to use in an iPhone application, in conjunction with a text to speech voice engine. Can someone point me in the right direction to see if there is a lipsynch tool that uses the audio volume to drive a morph? I need a simple open/closed movement of the mouth that reacts to the audio volume.



07-06-2009, 09:13 PM
Are you using a text-to-speech engine live on the phone? If so, what realtime 3D engine are you using? That will determine how you approach lipsync. Given the original text script, you should even be able to translate into basic phonemes and pass that on to the 3D engine.

On the other hand, if you're pre-rendering everything out of Lightwave and just want basic audio-driven lipsync, have you tried After Effects? It can generate keyframes based on waveform amplitude, might do the trick for you.

Tom Wood
07-06-2009, 09:25 PM
After looking into this some more, I see that there are what are called 'chatterbots' which emulate human conversation. Those are all presently text based.


They also have a text to speech module:


Odd that they didn't put the two together.

And then there's Siri:


You can tell it to do things, and it will. So, it's only a matter of time before they make one that talks back. That's all way above my ability, but I can see an add-on application that would provide animated avatars for the talking chatterbots.

"Just what do you think you're doing, Dave? "

07-06-2009, 09:44 PM
I think this has been done, at various levels, for some time. Microsoft attempted to make a go at interactive avatars (using both speech recognition and voice synthesis, albeit with pre-scripted responses) back in the late nineties. It didn't go quite so well for them... but I know other companies have released similar services more recently.

Actually, Microsoft is back at it - they demoed Project Natal (http://www.xbox.com/en-US/live/projectnatal/) at a conference earlier this year. That presentation included a live 3D character giving "natural" feedback based on vocal and body language cues.

There are plenty of limitations, of course, and everything I've seen in common use so far uses some form of pre-rendered assets (for example, Microsoft Agent required images for a given number of phonemes, looping idle animations, etc. in GIF format) or flash animation. Anything more dynamic than that is usually found in a game engine - and, AFAIK, those tend to be fairly scripted in terms of animated output, not quite as dynamic. I could be wrong, and by now I'm just jabbering on. :help:

Like you say, the pieces are there. I haven't seen anyone combine them in a particularly compelling way (though Natal comes close). :D

Tom Wood
07-06-2009, 11:08 PM
Cool stuff. Putting all the required processing power in a phone might be a challenge. The new iPhone does a lot more than the last one, but with a 15% smaller battery, people are having to recharge it every day just to keep up.

Edit: I think pre-rendered would work okay for this. My characters have predetermined poses with two versions for each - one where the mouth is moving and one where it isn't. If the amplitude of the audio could be monitored, then a crossfade between the mouth moving/not moving would fake it well enough.

07-07-2009, 09:46 AM
Back in 2000 I created Quipples: The Internet Game Show of Satirical Riddles (http://www.dreamlight.com/insights/09/welcome.html) as a proof of concept of my Autiton Intelligent Interactive Character Technology, which simulated intelligence through the programming of a virtual mind (http://www.dreamlight.com/insights/09/patent.html) and it also included an interactive 3D animated character component that used phoneme text strings to drive full lip sync to audio snippets assembled and streamed live over the Internet as you interact with the character. I programmed it in Macromedia Director's object-oriented Lingo language at the time. I've been swamped lately on The Autiton Archives - Fault Effect (& behind the scenes book) (http://www.dreamlight.com/insights/autitons/), but once that's done I may consider an iPhone app... based on Quipples, or maybe even DreamLight Verttice (http://dreamlight.com/insights/02/welcome.html).