Lip Sync Setup Guide

How to set up mouth movement for 3D characters in LocalNPC.

👄 What You Get in v1.0

LocalNPC has three mouth components. Only the first one moves a mouth with nothing else added:

1. AmplitudeLipSync: Opens and closes a jaw by the loudness of the NPC's voice. It needs no extra package. It does not make a mouth shape for each sound.
2. BlendShapeVisemeMapper: Turns viseme weights into shape key weights on your mesh. One sound can move several shape keys. It needs something that gives it viseme weights: your own code, or the uLipSync bridge.
3. ULipSyncBridge: Feeds the NPC's voice to the third-party uLipSync package and passes what uLipSync hears to the mapper. LocalNPC does not ship uLipSync. The bridge compiles with uLipSync installed. The code that feeds sound to uLipSync is tested with fake audio. The bridge itself has not run on a real character yet.

The QuickStart sample uses AmplitudeLipSync: the capsule NPC has a child "Mouth", a dark box that tips open with the loudness of the voice. The Shopkeeper and Kiosk scenes have no moving mouth.

When Windows has no voice for the NPC's language, the NPC answers in text only, nothing plays, and the mouth does not move.

🎨 3-Step Setup with AmplitudeLipSync

Step 1: Add AmplitudeLipSync

Select the NPC's GameObject (the one with the SpeechAudioQueue) or a child of it, for example the head:

Add Component, then search for Amplitude Lip Sync

Leave Audio Source (_audioSource, under Audio & Mesh Targets) empty. The NPC's voice plays one sentence on the NPC's AudioSource and the next on a second AudioSource that the SpeechAudioQueue adds when the scene starts. With the field empty, AmplitudeLipSync finds the SpeechAudioQueue on its own GameObject or a parent and reads both sources.

AmplitudeLipSync needs an AudioSource on its own GameObject. On a child, Unity adds one when you add the component; that one plays nothing, and you can leave it. If you assign an AudioSource that is not one of the NPC's voice sources, AmplitudeLipSync reads only that one.

Step 2: Give It a Jaw

The component uses the first SkinnedMeshRenderer it finds on its GameObject or its children, unless you assign one (_targetRenderer, under Audio & Mesh Targets). It then looks for a jaw-open blendshape on that mesh:

• First the name in _jawBlendShapeName, which is jawOpen by default.
• If that is not found: Jaw_Open, JawOpen, jaw_open, v_open, viseme_aa, MouthOpen, mouthOpen, mouth_open, Fcl_MTH_A, A.

If your mesh uses another name, type it into _jawBlendShapeName. If your mesh has no blendshapes, assign a jaw bone (_jawBone, under Bone Fallback) and set its closed and open rotations. The QuickStart "Mouth" is its own jaw bone: closed 0°, open 90° around X. With no matching blendshape and no jaw bone, the component does nothing.

Step 3: Connect to NpcAgent

You can leave the Viseme Driver slot of your NpcAgent empty: on start the agent looks for an AmplitudeLipSync on its own GameObject and then on its children. Or drag the AmplitudeLipSync into the slot. The agent uses the driver only to close the mouth when the NPC is interrupted and when a reply ends. The mouth movement itself comes from the voice. The slot only uses a component that implements IVisemeDriver, such as AmplitudeLipSync or ULipSyncBridge. Do not put a BlendShapeVisemeMapper in it; the agent ignores it there.

🧩 Using BlendShapeVisemeMapper

The mapper does nothing by itself. It needs viseme weights from your own code or from the uLipSync bridge (next section).

1. Select your character and add the component:

   Add Component, then search for Blend Shape Viseme Mapper

2. Assign your character's mesh in Skinned Mesh Renderer. If you leave it empty, the mapper uses the first SkinnedMeshRenderer on its GameObject or its children.
3. Choose the Viseme Standard: the 15-viseme set (sil, PP, FF, TH, DD, kk, CH, SS, nn, RR, aa, E, I, O, U) or ARKit-52 (jawOpen, mouthClose, mouthFunnel, mouthPucker, ...).
4. Click Auto-Detect BlendShapes from Mesh. The mapper looks for blendshapes named like the visemes of the chosen set. Upper and lower case do not matter, and these prefixes are also tried: v_, viseme_, vis_, ARKit_, blendShape., blendShape1. (for example viseme_aa). Visemes with no matching blendshape stay unmapped; you can pick a blendshape for each one by hand in the mapping list.
5. Feed it from your own code: call SetVisemeWeight(index, weight), SetVisemeWeights(weights) or SetVisemeByName(name, weight) on the mapper. A weight of 0 to 1 is read as a fraction; a weight above 1 is read as 0 to 100.

One sound, many shape keys

Each viseme in the mapping list has a Shape Keys list: the shape keys that this viseme moves together, each with its share of the full shape (0–1). When the list is empty, the viseme moves its single blendshape as above. From code: SetShapeKeys(visemeName, keys).

When several visemes move the same shape key, their parts are added and the sum is capped at Max Weight Ceiling. A shape key that is not on the mesh is skipped, with one warning in the Console.

The ARKit preset

For a head with ARKit shape key names, click Apply ARKit Preset (A, I, U, E, O, M/B/P). It sets the Viseme Standard to the 15-viseme set, replaces the shape keys you entered, and fills in this table (share of the full shape at a full viseme; L/R means the Left and the Right key, each at that value):

The other visemes, sil among them, move no shape key. These are starting values; tune them on your head by eye.

The mouthClose rule: ARKit mouthClose closes the lips while the jaw is open. When the mesh has both jawOpen and mouthClose and they are in the shape key lists, the mapper keeps mouthClose at or below jawOpen, so the lips do not go through each other.

🔌 The uLipSync Bridge (not tested on a real character)

uLipSync (com.hecomi.ulipsync) is a free third-party package that guesses vowels from the sound. LocalNPC does not ship it. Install it yourself through the Package Manager; when it is installed, ULipSyncBridge compiles. Without it, the bridge is not in the build and nothing else changes.

The bridge compiles with uLipSync installed (we compiled it with uLipSync 3.1.4). The code that feeds sound to uLipSync is tested with fake audio. The bridge itself has not run on a real character yet.

What the bridge needs:

• A GameObject without an AudioSource, for example a child of the NPC. Put uLipSync and ULipSyncBridge there. The bridge reads both voice sources of the NPC's SpeechAudioQueue (on its GameObject or a parent, or the one you assign in Speech Queue) and feeds them to uLipSync. On a GameObject with an AudioSource, uLipSync also hears that AudioSource itself, and the bridge logs a warning.
• A BlendShapeVisemeMapper on the same GameObject, or assigned in Viseme Mapper. The bridge passes each phoneme name and weight from uLipSync to the mapper's SetVisemeByName. Give your phonemes the mapper's viseme names (for example aa, not A); a name the mapper does not have moves nothing.
• A uLipSync calibration profile for the NPC's voice. uLipSync needs one to tell the vowels apart. LocalNPC ships none; make one with uLipSync's own tools.

The agent does not find the bridge by itself. To have the mouth close when the NPC is interrupted or a reply ends, drag ULipSyncBridge into the Viseme Driver slot of the NpcAgent.

🗿 Make a Head That Works

What the components above need from a head model:

• One mesh object with a SkinnedMeshRenderer (not a plain MeshRenderer). The mapper and AmplitudeLipSync each move the shape keys of one SkinnedMeshRenderer.
• The 12 shape keys, with these exact names: jawOpen, mouthClose, mouthFunnel, mouthPucker, mouthSmileLeft, mouthSmileRight, mouthLowerDownLeft, mouthLowerDownRight, mouthUpperUpLeft, mouthUpperUpRight, eyeBlinkLeft, eyeBlinkRight. The ARKit preset uses the first ten; AmplitudeLipSync uses jawOpen. LocalNPC has no blink component, so nothing moves the two blink keys yet.
• Teeth and tongue inside that mesh, moved by the same shape keys. A separate teeth or tongue object does not move with the mouth.
• Under 8,000 triangles. This is our own limit for the demo head, a suggestion, not a rule.

🧪 Tuning Speech Dynamics

AmplitudeLipSync, under Tuning Parameters:

• _gain (0.1–10): how far the jaw opens for a given loudness. The QuickStart "Mouth" uses 5.
• _noiseFloor (0.0001–0.1): below this loudness the mouth stays closed.
• _attackTime (0.01–0.3 s) and _decayTime (0.01–0.5 s): how fast the mouth opens and closes.
• _maxWeight (0–100): the largest blendshape weight applied.

In the BlendShapeVisemeMapper inspector:

• Smoothing Time (s) (0.01–0.5): how fast blendshapes move to a new weight. Lower is snappier.
• Max Weight Ceiling (0–100): the largest blendshape weight applied.
• Weight Multiplier (0.1–3.0), one per mapping: makes a single viseme stronger or weaker.

Rest State (Zero): When a reply ends or the NPC is interrupted, the agent resets the driver in its Viseme Driver slot and the mouth goes back to its closed position at once.