Performance Tuning & Optimization Guide
How to optimize LocalNPC latency, memory consumption, and frame stability.
⏱ End-to-End Latency Budgets: ≤ 1800 ms, design budget (not met on the one laptop measured)
The conversational pipeline consists of 6 sequential and overlapping stages:
| Stage | Component | Typical Time (Fast) | Acceptance Budget |
|---|---|---|---|
| Silence Detection | VoiceActivityGate (Hysteresis) | about 0.9 s by design: the turn ends after 900 ms below the end threshold. Replayed offline (not in the app): 0.94 to 1.10 s after the last word | none: a fixed wait of about 0.9 s, before the model time below |
| Speech-to-Text | Whisper tiny.en, in-process | measured: 0.62 to 0.90 s for a 3 s clip, 0.61 to 1.07 s for an 11 s clip (see the note below) | ≤ 350 ms |
| LLM Prefix Lookup | KV Cache Hit | not measured | ≤ 50 ms |
| LLM 1st Sentence | Qwen2.5-0.5B (8–14 tokens) | not measured | ≤ 450 ms |
| Text-to-Speech | Windows system voice (1st Sentence) | not measured | ≤ 400 ms |
| Audio Pre-roll | SpeechAudioQueue Scheduling | not measured | ≤ 150 ms |
| Total Turn Latency | End-of-Speech → First Audio | not measured for Fast. Balanced: about 2 s after the 0.9 s wait (2.04 to 2.25 s on one laptop, five voice turns; see below) | ≤ 1800 ms, design budget (not met on the one laptop measured) |
What is measured in this table: the speech-to-text row, and the silence-detection row offline: recordings of one voice with fan noise, replayed through a copy of the microphone chain, not measured in the app. Speech-to-text starts only after the silence wait, so its time and the model times come on top of it. Speech-to-text was measured on one laptop (Intel Core i7-8750H, 15.8 GB RAM, Windows 11), tiny.en model, 4 threads, on battery, one or two runs per figure. On that laptop the speech-to-text step alone is over its budget, and a 3 s clip costs about as much as an 11 s clip. The total row was measured with the Balanced preset, on the sample scenes' HUD clock (see "Two clocks" below); the other rows of the "Typical Time" column are not measured. Reply time: Typically about 3 seconds after you stop talking: a 0.9 second end-of-speech wait plus about 2 seconds (measured 2.04 to 2.25 s on one laptop, Balanced preset, five voice turns). It depends on your hardware, and we have not measured other machines yet.
🧵 Thread Configuration (threadCount)
In NpcRuntimeConfig:
threadCount = 0(Auto): Usesmax(2, min(physical cores - 2, 6))threads. The two cores left out are headroom for Unity's main thread and render thread.- Dedicated CPU Allocation:
LocalNPC-Inference: Runs Whisper STT and Llama LLM sequentially. The Windows voice speaks each sentence on a thread-pool thread. Speech-to-text runs in-process on this thread, so a transcription waits behind a reply that another NPC is generating.- There is no separate audio thread in v1.0: microphone audio is processed on Unity's main thread, once per frame.
- Rule of Thumb: leave
threadCountat0(Auto). Other values were not measured.
🪟 Context Window & History Trimming
In NpcRuntimeConfig:
contextLength = 2048: The default. How many conversational turns fit has not been measured yet.- Memory Impact: Memory per context token is not measured yet. On file: about 473 MB for one NPC and about 46 MB for each extra NPC with the 0.5B model (measured in the Editor on one PC, one run each). The loaded speech model adds about 135 MB of working set and about 650 MB of private bytes (measured on one laptop (Intel Core i7-8750H, 15.8 GB RAM, Windows 11),
tiny.enmodel, 4 threads, on battery, one or two runs per figure). Setting context to 8192 unnecessarily consumes RAM and increases prompt evaluation times. - Pairwise Trimming Rule: LocalNPC's
ConversationSessionautomatically preserves the system prompt + few-shot exemplars, and drops the oldest user/assistant pairs first, guaranteeing that conversational chat templates never become corrupt.
🔍 Profiling with Diagnostics Window
Open Tools > LocalNPC > Diagnostics Window during Play Mode:
- Live Latency Breakdown: A bar chart of the STT, LLM TTFT and TTS timings of each turn. Its clock starts when the pipeline gets the turn, so the end-of-speech wait of about 0.9 s is not in it. The window compares each turn with a target of 2500 ms to the first audio, after the end-of-speech wait; a turn above it is shown in amber as information, not as an error. On the one laptop measured (Balanced), turns took about 2 s on the HUD clock, so they show within this target. Slower PCs may show above it; that is information, not an error.
- Generation Throughput: Measures tokens-per-second (tok/s). For comparison, the Fast preset (0.5B) gave 50.4 to 55.8 tok/s in a player build without a window, and 28.4 and 36.8 tok/s in a windowed player; the default Balanced preset (1.5B) gave 21.0 and 23.7 tok/s in a player build without a window, for a 12-token reply (windowed: not measured yet) (one PC (Intel Core i7-8750H, 16 GB RAM, Windows 11), Unity 6000.3.10f1, one run per figure).
- RAM (MB): Graph of the peak resident RAM of each turn. The RAM card compares it with the target of the preset the agent runs: Fast has a target of 800 MB; Balanced has no RAM target.
- Two clocks for "time to first audio". The sample scenes' HUD shows Time to first audio with its own clock: for a voice turn it starts when the NPC stops listening, after the same 0.9 s wait, and stops when the first sentence starts to play. The Diagnostics window uses the pipeline's clock. The two are measured at different points, so their numbers can differ; neither includes the 0.9 s wait.