Recently I’ve been working on an animation for the pharma division of Harte-Hanks. It’s in our classic MZ-stylee and it’s been fun doing some old-school 2d Flash animation. Zoe’s given me some great illustrations to work with and Charlie’s imagination has been running riot with karate-chopping leaps and flabby men on treadmills. Lots of tweens, lots of Graphic symbols, lots and lots of keyframes. I’ve also gotten to play with CS4′s new IK bones for some of the sequences too but that’s a whole other post.
From the outset though, I’ve been procrastinating about lip-syncing the narrator’s voice-over with the character. My rationale was that any edits to the timing would mean re-doing work… always fun, so best leave it til last, but really it just seemed like it’d be an arduous time-consuming manual task. In all the years I’ve been doing animation in Flash I’ve never found an easy way of doing this. Animation is a craft and it seems the only solution to most tasks is put some time and love into it. That said, code is your friend (and a cruel, cruel mistress) that has often helped avoid the monkey work.
So I went looking for a pre-existing solution… First port of call, as always, was “The Animator’s Survival Kit” by Richard Williams (if you don’t own it, get it!). Then some Googling that rapidly disappeared down the rabbit hole of visemes, phonemes and natural language simulation dissertations. Whew! No easy answers there either.
At the last Bristol FUG, I had a good chat with master-animator Dave Cropley (from Evans & Finch) about how he does it. He pointed me towards software called Magpie that seems to be the standard in the 3d animation world. Nothing seemed to exist in the Flash/ActionScript world though so I decided to roll my own.
Here’s how I did it…
Step 1
Mac OSX has a little-known (and well-hidden) application called Repeat After Me that got me a lot of the way. First off open up Repeat After Me and paste in the text of your speech. Then go to ‘Sound’ > ‘Attach…’ and attach an .aif sound file. After a couple of seconds you get a wave-form representing the loaded audio.
Step 2
Next, hit “To Phonemes” and then “Build Graph”. Magic.
The Phonemes field gets filled with a stream of gibberish like this:
_h1AY, ~2AYm _d1AAktAXr _h1EHlIXn. ~lEHt ~mIY _t1EYk ~yUW ~2AAn ~AX _f1AOrtIY _f1AYv _s1EHkAXnd _t1UHr ~AXr2AWnd ~DAX _h1AAr ~t2EY _h1AENks _w1IHN.
We’ll deal with how to use that later. Your graph should look something like this:
Have a look a little closer and you’ll see the generated phonemes have been almost matched to the wave-form. The grey areas are the spaces between words and each band of colours shows a specific phoneme.
Step 3
Now the fiddly part. Repeat After Me is pretty good at matching it but it will need a human touch here. Go through and drag the edges of colours so that the grey areas match the flat lines and you’ll see the wave-form start to tie-in with the bands of colour. This can be pretty fiddly so make sure you hold Shift and click the edges of the area to lock/unlock what you’re happy with.
Step 4
Hit ‘To Phonemes’ again and your tweaked graph will feed in to the Phonemes field and give you your timing data.
This is turning into a mammoth post so tune in to the next one for the ActionScript translation of this to an automated lipsync!
[EDIT: This post is continued in Adventures in Lip-sync: Part 2]
Tags: actionscript, Animation, Flash, lip-sync