Troubleshooting Playbook

This playbook covers common symptoms, root causes, and verified fixes when integrating, tuning, or deploying LocalNPC.

Quick Reference Diagnostic Matrix

1. Native & Platform Issues

Error NativeLibraryMissing (a llamalib_win-x64_*.dll could not be loaded)

• Cause 1: 32-bit Architecture. The native libraries are compiled strictly for 64-bit Windows (x86_64). If your Unity Build Settings are set to 32-bit (StandaloneWindows), Windows will fail to load the DLL.
  • Fix: In File > Build Profiles, set Target Platform to Windows 64-bit (StandaloneWindows64).
• Cause 2: The file is not in the build. LocalNPC loads its native libraries from <Game>_Data/Plugins/x86_64/ only; it does not search next to the .exe or on the system path.
  • Fix: Unity copies all eight DLLs into that folder during a normal build (two llamalib_win-x64_*.dll files for the language model, ggml-base.dll, ggml.dll, whisper.dll, ggml-cpu-haswell.dll and ggml-cpu-x64.dll for speech-to-text, and localnpc_winspeech.dll for the voice). If you use custom build scripts or move files after the build, keep all of them there.
• Cause 3: Missing Visual C++ Redistributable.
  • Fix: Ensure the target machine has the Visual C++ 2015–2022 x64 Redistributable installed.

"Windows blocked this file" (Windows error 4551)

• Message: Windows blocked this file: native library '<file>' at '<path>' was not allowed to load (Windows error 4551). This is usually caused by Windows Application Control (Smart App Control) or by antivirus software.
• Cause: Windows Application Control (Smart App Control) or antivirus software blocked one of the native libraries. The libraries are not signed.
• What to try, in this order:
  1. Move the game out of the Downloads folder or a synced folder (OneDrive, Dropbox and similar).
  2. Add an antivirus exclusion for the game folder.
• Never turn off Smart App Control, and never tell your players to.
• For a speech-to-text library the message goes on with: Speech input is off. Text input still works. Ask the player to allow the file in Windows Security. Typed input keeps working. If only ggml-cpu-haswell.dll is blocked, LocalNPC loads ggml-cpu-x64.dll instead and logs a warning that names the blocked file.
• For the voice library (localnpc_winspeech.dll) the warning ends with The NPC voice is off; replies are shown as text. The NPC answers in text only.
• Not tested: none of this advice has been tried against a blocked LocalNPC library. During development, Smart App Control blocked ggml.dll once and allowed the same file in the next run. In later test runs with Smart App Control on, nothing was blocked. The decision depends on the reputation Windows gives a file and may differ on other PCs.

Speech Input Is Off (Text Input Still Works)

Speech-to-text runs inside the game's own process, through the five whisper.cpp 1.9.5 DLLs in Plugins/x86_64. No .exe is started. The speech model is the speech-to-text entry of the model catalogue.

When the model or a library cannot be loaded, NpcAgent raises OnError once with the real error code. The message says what is wrong and what to do, and contains the sentence "Speech input is off. Text input still works." The agent goes to Idle, typed input (SayAsync) works, and StartListening() reports an error and does not start the microphone.

• ModelMissing: the speech model is not on disk. Download it in Tools > LocalNPC > Model Manager.
• ModelCorrupt: the file has another size or SHA-256 than its catalogue entry (for example a download that stopped half way). Delete it and download it again in the Model Manager. LocalNPC checks the file before every load and never loads a file that does not match.
• NativeLibraryMissing: a DLL is missing, or Windows blocked it. See the two entries above.
• AbiMismatch: a DLL has another SHA-256 than the package was tested with, or another plugin in the project has a file with the same name (ggml.dll, ggml-base.dll or whisper.dll). Restore the package files, or remove the other copy, and start the application again.

LocalNPC does not try again by itself after you fix the cause. Leave Play Mode and enter it again (in a game: restart it).

The NPC Answers in Text Only (No Windows Voice)

The NPC speaks with a Windows system voice: a OneCore voice, with SAPI 5 as the last fallback. LocalNPC downloads no voice. For en-US it tries Microsoft Mark, then Microsoft Zira, then Microsoft David, then any other en-US voice, then a voice of another English region (en-GB, en-AU, ...), then a SAPI 5 en-US voice. When it finds none, every reply is still shown (OnSentenceSpoken fires for each sentence), nothing is heard, and the Console shows one warning:

No Windows voice for language 'en-US' is installed (none of [Microsoft Mark, Microsoft Zira, Microsoft David], no other voice of that language or of 'en', no SAPI 5 voice). Add one in Windows Settings > Time & language > Speech > Manage voices > Add voices. The NPC voice is off; replies are shown as text.

• Fix: On the PC that runs the game, open Settings > Time & language > Speech and add a voice for the NPC's language under Manage voices > Add voices. Then start the game again; LocalNPC looks for a voice once. The Manage voices path was checked on Windows 11, not on Windows 10.
• The Windows 11 "Natural" voices of Narrator are not available to LocalNPC (checked on one PC during development), so adding one does not give the NPC a voice.
• The voice chosen as the default in Windows Settings does not change the NPC's voice: LocalNPC takes the first installed voice of the language profile's list (measured on one PC: with Microsoft David as the Windows default, LocalNPC used Microsoft Mark).
• In the Editor, the Setup Wizard's environment step shows the row Windows voice installed as a warning when it finds no voice for en-US.
• In the sample scenes, the HUD shows one line under the state badge: No voice found - showing text only.
• Known issue: an en-US NPC on a PC without an en-US voice speaks with a voice of another English region, for example Microsoft Hazel (en-GB), with that accent. This was checked with test voice lists only, not yet on a PC that has only English UK voices.
• When the voice library itself cannot be loaded, the warning names localnpc_winspeech.dll and the reason instead, and ends with the same sentence. See the entries above.

Restart Unity After Replacing the DLLs

Once loaded, the speech-to-text libraries and the voice library stay loaded for the life of the process. Leaving Play Mode, a script reload and Unload() free the speech model, but not the libraries. So after a package update, or after you replace a file in Plugins/x86_64 by hand, close Unity and open it again; until then the process still uses the old files.

CPU Without AVX2, and LOCALNPC_FORCE_NOAVX=1

• How the library is chosen: LocalNPC asks Windows whether the CPU has AVX2. If yes, it loads llamalib_win-x64_avx2.dll. If no, it loads llamalib_win-x64_noavx.dll and logs the warning This CPU does not report AVX2. Loaded 'llamalib_win-x64_noavx.dll'; text generation will be slow.
• Speed of the fallback: 5.4 and 6.4 tokens per second, against 52.7 and 55.8 with the AVX2 library in the same two builds (0.5B model of the Fast preset, Windows player, one PC, one run each; with the 1.5B model of the default Balanced preset, not measured yet). The fallback was forced with the environment variable on a CPU that has AVX2. A CPU that really lacks AVX2: not measured.
• Assumption: the AVX2 library also uses FMA, F16C and BMI2. Windows has no query for FMA or F16C, so LocalNPC assumes all three are present whenever Windows reports AVX2. A CPU with AVX2 but without one of them would crash inside the AVX2 library. This has not been observed.
• Workaround: set the environment variable LOCALNPC_FORCE_NOAVX=1 before the game starts. CPU detection is skipped, the noavx library is loaded, and a warning is logged.
• Speech-to-text chooses by itself: it loads ggml-cpu-haswell.dll when the CPU can run it, and otherwise ggml-cpu-x64.dll with the warning Loaded 'ggml-cpu-x64.dll' instead; speech recognition will be slow. Speech-to-text with ggml-cpu-x64.dll, and on a CPU that really lacks AVX2: not tested and not measured.
• LOCALNPC_WHISPER_CPU_BACKEND: LocalNPC sets this environment variable inside its own process, to remember which of the two CPU libraries is in use. It is used internally. Do not set or change it.

Editor Crashes on Exiting Play Mode

• Cause: Native memory allocations (such as Llama context handles or Whisper state) were not released before Unity reloaded the AppDomain.
  • Fix: Always call Dispose() or unload models before domain reload. LocalNPC wraps all native pointers in SafeHandle classes that automatically hook into AssemblyReloadEvents.beforeAssemblyReload. Never bypass these wrappers with raw IntPtr.

Works in Editor, Fails in Standalone IL2CPP

• Cause: Unity's IL2CPP bytecode stripper removed serialized types or native P/Invoke bindings.
  • Fix 1: In Project Settings > Player > Other Settings, set Managed Stripping Level to Minimal or Low.
  • Fix 2: Ensure link.xml is present in your project:

    <linker>
      <assembly fullname="LocalNPC.Runtime" preserve="all"/>
    </linker>

The Sample Scene Is Pink

• Cause: the project uses HDRP. HDRP is not supported; the sample scenes draw in the Built-in Render Pipeline and in URP.
  • Fix: Try the samples in a project made from the 3D or the Universal 3D template, as described in Installation & System Requirements.

DirectoryNotFoundException in Library\PackageCache

• Keep your project folder path short, for example under 100 characters, because Unity's own packages can fail on long Windows paths.

2. Audio & Voice Activity Detection (VAD)

Error MicrophoneUnavailable

NpcAgent raises this through OnError when it is listening and the microphone cannot be used. It is raised once per failure, not every frame. The agent stays in the Listening state. A missing microphone and a failed start are tried again every 2 seconds. If the NPC still does not hear you after the cause is removed, call StopListening() and then StartListening(). The message tells the three cases apart:

• "No microphone is connected" / "Microphone '...' is not connected". Connect a microphone, or enable it in the Windows Sound settings.
• "... could not be started". Unity could not start recording on the device. Another application may be using it; close that application.
• "... started but delivers no audio". Recording started but no sound arrived in the first 3 seconds. Check, in this order:
  1. Windows Settings > Privacy & security > Microphone: microphone access and access for desktop apps must be on.
  2. The device is not muted (hardware mute switch, or the input volume in the Windows Sound settings).
  3. No other application is holding the device.

LocalNPC cannot tell these three causes apart from silence alone, so it never raises MicrophonePermissionDenied on Windows.

The NPC Does Not Hear Me

If there is no error and the NPC never reacts to speech:

• Listening was not started. Tick Listen On Start on the NpcAgent, or call StartListening() after PrewarmAsync(). The state must be Listening.
• Another NPC owns the microphone. Only one NpcAgent can listen at a time. A second agent logs the warning "has no voice input: the microphone is already used by ..." and gets no voice input. Call StopListening() on the first agent, then StartListening() on the other.
• The NPC is speaking. The microphone is ignored while the NPC speaks and for a short moment after, so that it does not answer its own voice. Speaking over the NPC does not interrupt it; call Interrupt() from code (for example from a button).
• Use Built-In Microphone is switched off on the NpcAgent. Then the agent expects your own code to capture audio and send it to the pipeline.

First Word of Every Utterance Missing

• Cause: When a player starts speaking, the VAD needs a short stretch of speech before it confirms that speech has started (180 ms with the microphone that NpcAgent starts). Without a pre-roll buffer, audio from that initial onset window is discarded.
  • Fix: The microphone that NpcAgent starts keeps the last 500 ms of audio and puts it in front of each utterance. This value is fixed in the code; NpcRuntimeConfig has no setting for it. If you capture audio yourself with a VoiceActivityGate, set its PreRollMs (default 300).

"The NPC Cuts Me Off Mid-Sentence"

• Cause: Natural human speech contains brief micro-pauses (e.g., between clauses or when thinking). If minSilenceMs is too aggressive (e.g. 150 ms), the VAD assumes the user has finished speaking prematurely.
  • Fix: The microphone that NpcAgent starts ends an utterance after 900 ms of silence. This value is fixed in the code; NpcRuntimeConfig has no setting for it. If you capture audio yourself with a VoiceActivityGate, raise its MinSilenceMs (default 350).

NPC Answers Questions Nobody Asked (Hallucinations)

• Cause: Whisper models can hallucinate repetitive phrases (e.g., "Thank you.", "Subtitles by...") when fed background static, air conditioner hum, or pure silence.
  • Fix: There is no setting for this. The pipeline drops a transcript that is empty or one character long, that is only a tag in brackets (for example [BLANK_AUDIO]), or that is one of a short list of phrases such as "thanks for watching", "subtitles by", "you" and "the end". A transcript of only "thank you" or "bye" is dropped only when Whisper reports that the turn holds no speech (a no-speech probability of 0.5 or more); otherwise the NPC answers it.
  • LocalNPC also includes SttPostProcessor.cs, with MaxNoSpeechProbability (default 0.6) and MinCharacters (default 2). The pipeline does not use this class yet; only the Speech to Text Check window does.

NPC Talks to Itself in an Echo Loop

• Cause: Audio from the game speakers enters the open microphone. The STT transcribes the NPC's own words, and the LLM responds to itself.
  • Fix 1: LocalNPC features built-in Acoustic Echo Gating. While the NPC is speaking (and for a 150 ms acoustic tail after speech ends), the microphone input gate is clamped.
  • Fix 2: Recommend players wear headphones or a headset. For a kiosk, use a headset or a directional microphone aimed at the visitor, and keep the speakers no louder than needed. Push-to-talk is not a fix in v1.0: PushToTalkController is not connected to the microphone (planned for v1.1).

3. Latency & Performance

"It Takes Forever to Respond" (> 2.5 seconds)

Open Tools > LocalNPC > Diagnostics Window during Play Mode to view the exact per-stage latency breakdown:

• If t_transcript is > 600 ms: This can be normal today. Measured for the speech-to-text step alone: 0.62 to 0.90 s for a 3 s clip and 0.61 to 1.07 s for an 11 s clip (one laptop (Intel Core i7-8750H, 15.8 GB RAM, Windows 11), tiny.en model, 4 threads, on battery, one or two runs per figure). A short clip costs about as much as a long one. LocalNPC already uses the tiny.en Whisper model; there is no smaller model to switch to. Much higher values mean the CPU is oversubscribed, or the transcription waited behind another NPC's reply.
• If t_prefill is high on every turn (no normal figure was measured): the prompt cache may not be working. See the section below.
• If t_first_pcm is high (no normal figure was measured): the Windows voice is slow on this PC. There is no voice quality setting to change.

Latency Doubled After Changing the System Prompt

• Cause: Prefix Cache Invalidation. LocalNPC computes the prompt cache for the system prompt once, in PrewarmAsync, and reuses it on every turn. If your prompt contains dynamic elements that change every turn:

  // BAD: Invalidate cache every single second
  "You are a wizard. The current time is: " + DateTime.Now.ToString();

  The LLM must re-process the entire prompt from scratch on every turn. How much slower this is has not been measured yet.
• Fix: Keep system prompts completely static and byte-identical. Pass dynamic gameplay variables (player gold, quest state) as part of the user turn, not the system prompt.

Frame Rate Hitches Every Few Seconds

• Cause: GC Allocations in Audio Stream.
  • Fix: Ensure no code allocates arrays or strings inside Update(), OnAudioFilterRead(), or audio processing loops. LocalNPC creates its audio buffers (AudioRingBuffer, resamplers, VAD tensors) once and reuses them, so look at your own code first. Check the Unity Profiler CPU usage sorted by GC Alloc.

Audible Gaps Between Sentences During Speech

A pause of 300 ms after a sentence that ends with ., ! or ? is intended: the Windows voice adds it so that sentences do not run together. A long sentence that is cut before its end gets no such pause.

• Cause: Polling AudioSource. If speech synthesis finishes a sentence and waits for AudioSource.isPlaying == false before starting the next clip, there is a short silent gap (not measured).
  • Fix: Ensure SpeechAudioQueue is configured to use AudioSettings.dspTime. This queues clips with sample-accurate DSP hardware scheduling, delivering seamless, natural sentence transitions.

4. LLM Generation Quality & Formatting

NPC Speaks <|im_end|> or *smiles warmly* Out Loud

• Cause 1: Missing Stop Sequences. The model emits ChatML framing tokens because stop sequences were not registered.
  • Fix: Ensure <|im_end|>, <|endoftext|>, and Player: are listed in NpcPersonality.stopStrings. The first two are there by default.
• Cause 2: Markdown / Action Tags. The LLM includes roleplay actions like *sighs* or (laughs).
  • LocalNPC v1.0 does not remove these before the voice reads the reply. Fix: tell the NPC in its personality not to write actions, for example: "Never describe actions. Never use asterisks or brackets."

Garbled Non-ASCII or Unicode Characters in Replies

• Cause: Multi-Byte UTF-8 Fragmentation. LLM token streams emit text token-by-token. A single Unicode character (such as an emoji or accented character é) consists of 2 to 4 raw bytes. If a token boundary falls between these bytes and is converted directly to a C# string, it yields the replacement character ``.
  • Fix: LocalNPC uses LlamaTokenStream.cs, which buffers incomplete byte sequences and only emits completed UTF-8 characters to the UI and TTS.

NPC Contradicts Itself or Gets a Fact Wrong (Limits of a Small Model)

• Cause: the 0.5B model of the Fast preset is small. It makes mistakes that a better prompt does not fully remove. The 1.5B model of the default Balanced preset is larger; whether it makes fewer of these mistakes is not measured yet.
• Example from a five-turn test chat in the Editor (0.5B model, Fast preset): asked "Is the north road safe to travel?", the NPC answered "Yes, it's a bit dangerous on the north road due to bandits." The fact (bandits) came from the system prompt and was right; the "Yes" contradicts it.
• How often this happens: not measured.

5. Multi-Agent & Memory

Second NPC Causes Out-of-Memory (OOM)

• Cause: a separate copy of the language model in memory for every NPC in the scene.
  • Fix: LocalNPC uses NpcServiceLocator.cs with reference-counted model sharing. Multiple NpcAgent components in the same scene automatically share the same underlying LLM and Whisper model weights, each maintaining only its own KV cache (about 46 MB per extra NPC with the 0.5B model, measured in the Editor on one PC, one run).

A Second NPC Waits Before It Replies

• Cause: NPC replies take turns. Only one reply is generated at a time across all NPCs. Each NPC keeps its own conversation cache, but a second NPC's reply starts only after the first one ends.
• Measured in the Editor: a second NPC waited 1354 to 1454 ms behind a 64-token reply and 4121 to 4866 ms behind a 200-token reply.
  • Fix: when several NPCs can talk at once, keep replies short with a low max-tokens setting.
• Voices do not wait: two NPCs in one scene do not wait for each other, so if two are asked at the same time they speak at the same time; you must sequence them in your own game code.

Only One NPC Can Listen at a Time

Only one NpcAgent can listen to the microphone at a time. Call StopListening on one before StartListening on another.

Unload() Returns but the Model Stays Loaded

• Cause: if a native call never returns, Unload() gives up after 4 s and logs an error. The model stays loaded, later work waits behind the stuck call, and the memory is freed only when the application exits.
• Not reproduced: this is known from reading the code, not from a run.

6. Known Issues in v1.0

Known limits of this version, and what to do about each one.

• Very short phrases. A very short phrase can still be dropped when Whisper reports no speech for the turn; the phrases "you" and "the end" are always ignored. If a short reply gets no answer, say a little more, or send it as typed text.
• VoiceActivityGate event order. If you use this class in your own code: it can raise OnUtteranceCancelled just before OnUtteranceCompleted; treat Completed as the result.
• Loud steady noise (fan, air conditioner). The microphone follows the level of steady noise (tested offline with recordings of one fan, not in the app), but speech must be about 8 dB louder than the noise to be heard, and a voice turn ends after 12 s at most. If the NPC answers late or not at all near loud noise, and the Speech Level meter (Inspector of MicrophoneCapture, in Play mode) shows more than about 20 % while nobody speaks, lower the microphone input volume in Windows (Settings > System > Sound > Input) or move the microphone away from the noise.
• Reply time. Typically about 3 seconds after you stop talking: a 0.9 second end-of-speech wait plus about 2 seconds (measured 2.04 to 2.25 s on one laptop, Balanced preset, five voice turns). It depends on your hardware, and we have not measured other machines yet. On a 2018 laptop the speech-to-text step alone took 0.6 to 0.9 s for a 3 s clip.
• Older CPUs. Tested on CPUs with AVX2; on older CPUs a slower library is used, which was not tested on such a CPU.
• Kiosk models. In StreamingAssets mode, the language model in the download folder is used first if one is there. For a kiosk, keep only the bundled copy.
• Microphone unplugged or switched. This is tested with automated tests only. If voice input stops, call StopListening and then StartListening.
• One listening NPC. Only one NpcAgent can listen to the microphone at a time. Call StopListening on one before StartListening on another.
• Kiosk sample. The Kiosk sample was tested with loudspeakers; a headset was not tried.
• Speech recognition out of memory. If it runs out of memory, voice input stops until you call PrewarmAsync again. Typed input keeps working.
• Voice field of NpcPersonality. NpcPersonality.voice is not used in v1.0; set the voice with NpcAgent.SystemVoice.
• StopAfterSentence. After StopAfterSentence, a sentence that was already being prepared can still be heard.
• ARKit preset. The ARKit preset has not been tested on a real head yet.
• uLipSync. uLipSync is optional and not included. Put uLipSync and ULipSyncBridge on a GameObject without an AudioSource, for example a child of the NPC. Then drag ULipSyncBridge into the NPC's Viseme Driver slot. LocalNPC ships no uLipSync calibration profile; make one with uLipSync's own tools. See Lip Sync Setup.
• Narrator voices. The Windows 11 "Natural" voices of Narrator are not available to LocalNPC (checked on one PC). Add a normal voice in Settings > Time & language > Speech.
• Other platforms. On macOS, Linux and other platforms LocalNPC does nothing. But if your own code calls MicrophoneCapture.StartCapture or LocalNpc.EnsureModelsAsync there, the microphone still opens or the download still starts. Do not call them on these platforms.
• WebGL. WebGL builds do not compile: the microphone code uses Unity's Microphone class, which WebGL does not have (found by reading the code; no WebGL build was tried).
• Smart App Control and Burst. When the optional Inference Engine package (com.unity.ai.inference) is installed, Windows Smart App Control can block the file that Unity's Burst compiler builds in the Editor (error 4551 in the Console; seen on one development PC). LocalNPC v1.0 does not use the Inference Engine. Workaround, in the Editor only: turn off Jobs > Burst > Enable Compilation. Do not turn Smart App Control off.

Still Need Help?

If you encounter an issue not covered in this playbook:

1. Export a diagnostics report: after a Play session with the NPC, open Tools > LocalNPC > Diagnostics Window and click the Export CSV button.
2. Check the generated LocalNPC_BuildManifest.json for checksum mismatches.
3. Email us, as described below.

Getting help

Email localnpcsupport@gmail.com. Please include:

• your Unity version (for example 6000.3.10f1),
• your Windows version (for example Windows 11 23H2),
• the Console log (in the Console window's menu, Open Editor Log; for a build, its player log),
• the Diagnostics window export (the CSV file from step 1),
• whether the problem also happens in the 01_QuickStart sample scene.

We are a small team, so a reply can take a few days.