ElevenLabs connected voice-enabled NPCs to Unity with 10 minutes of setup

A voice-enabled NPC in Unity usually starts as an entertaining prototype and ends up as a familiar tangle of microphone handling, networking, audio playback, and synchronization with game state. ElevenLabs has bundled these components into a dedicated Unity SDK for ElevenAgents. The player approaches a character and speaks aloud; the character answers by voice, knows about events in the scene, and invokes game code when necessary.

The company says the starter example can be running in about 10 minutes. That is not a promise of a release-ready character, but it is a useful way to quickly test the core product hypothesis: whether players actually want to talk to an NPC instead of selecting prewritten responses.

How the conversation works

The developer creates a voice agent in the ElevenLabs dashboard and passes its identifier when starting a session. The SDK opens the connection, configures the audio devices, and selects the appropriate transport. WebSocket is used in the Editor and desktop builds, while WebGL supports either WebSocket or WebRTC.

A basic scenario works like this: a trigger is placed around the character. When the player enters its area, the game calls StartSessionAsync; when the player leaves, it calls EndSession. By default, the voice is played as ordinary two-dimensional audio, but the developer can provide a custom AudioSource so the line comes from the character's position.

The SDK also returns the current response volume on a scale from 0 to 1. That is already sufficient for rough mouth animation, an illuminated indicator, or model movement. It does not replace full lip synchronization, but it makes it possible to build a clear prototype without a separate facial-animation pipeline.

One agent, multiple characters

There is no need to create a separate agent for every NPC. When a session starts, the game can pass dynamic variables such as the character's color, profession, or mood. These values are inserted into the system instructions and opening line. More substantial differences can be configured through temporary voice and agent-setting overrides, provided they have been permitted in advance in the security settings.

Imagine two innkeepers in the same scene. Both use the same basic conversation script, but the first receives the variables “calm” and “friendly,” while the second receives “irritated” and “suspicious.” The savings extend beyond the number of configurations: shared behavioral rules can be changed centrally while preserving differences between characters.

Game events are passed through SendContextualUpdate. If the player finds the basement key, the agent learns about it without an artificial line spoken on the player's behalf. Feedback works through client tools: the agent can request that a door be opened, a quest started, or an item granted, and a registered handler performs the action and returns the result.

This is an important architectural boundary. The model proposes an action, but the game executes it. The handler must therefore independently validate permissions, scene state, and parameter admissibility. A text model must not be allowed to directly change inventory or progression merely because it produced a convincing function call.

What to consider before release

The Editor, Windows, macOS, Linux, and WebGL are listed as supported platforms. iOS, Android, and XR are currently in beta. Mobile devices require a separate microphone-permission request. WebGL requires WebAssembly.Table to be enabled; browser playback also has limitations: some AudioSource settings do not carry over to the Web Audio API, and WebRTC ignores the supplied OutputAudioSource.

The main limitation concerns security. For an initial local test, ElevenLabs suggests disabling authentication and connecting using the agent identifier. This must not be done in a published game. The developer's server should obtain a short-lived signed URL and pass it to the client. The ElevenLabs API key must not be included in the build, as it could be extracted and used at your expense.

The practical conclusion is straightforward: the SDK is suitable for rapidly building a vertical prototype of a conversational character. Start with one NPC and one verifiable action, such as opening a door. Then measure response latency, session cost, error rate, and the share of players who actually use voice. If the mechanic works, add memory, animation, and new tools—but keep final validation of in-game actions inside your own code.