AWS teaches voice AI agents to start speaking from the first sentence with Kokoro-82M

A user asks a voice agent a question and waits. The answer is already partly generated, but there is no sound: text-to-speech integrations usually wait for the entire text to finish. The model produces one sentence in 600 ms, while a long answer takes about five seconds in total. For that whole time, the client listens to silence. In its blog, AWS showed how to close this gap: send each sentence to the Kokoro-82M synthesizer on Amazon SageMaker AI as soon as the agent finishes it.

A small model, one GPU and two routes

Kokoro-82M has 82 million parameters, which is small enough to run on a single mid-range GPU instance. The model runs in a custom container on a SageMaker AI real-time endpoint, using the ml.g5.xlarge instance type. The container exposes two routes: POST /invocations for a streamed response to a single complete input, and WS /invocations-bidirectional-stream, where one connection carries many messages in both directions. The container must declare the Docker label com.amazonaws.sagemaker.capabilities.bidirectional-streaming=true.

Deployment comes down to two AWS CDK commands. `uv run speech-infra deploy kokoro-82m` brings up the endpoint (IAM roles, a weight cache in S3, the container image, the model and the configuration). `uv run agent-infra deploy --bedrock-model-id <id>` deploys a Strands agent in the Amazon Bedrock AgentCore environment. Order matters: the agent calls an endpoint that must already exist. Python 3.12+ is required, because the HTTP/2 client ships prebuilt wheels only for those versions. The authors also include a warning: the example pins an old PyTorch version with a known vulnerability, PYSEC-2026-2286 (fixed in torch 2.10.0), and it must be updated before any use beyond a proof-of-value.

Three pitfalls of bidirectional streaming

The first pitfall is the WebSocket frame type. SageMaker delivers every client message as a binary frame. A typical Starlette handler calls receive_text(), reads message["text"] and crashes with a KeyError. If a generic exception handler sends str(e) back to the client, the client sees the error "'text'", which looks like a missing field rather than a framing problem. Meanwhile the endpoint reports InService: it accepts connections and answers every message, just with an error. The fix is to accept both frame types and, if there is no text field, decode the bytes as UTF-8.

The second is premature session closure. After sending the audio and the synthesis_complete control frame, the container must return to receive() and wait for the next message. It should close the connection only on an explicit {"type": "close"}. Otherwise every sentence needs a new connection, and the latency gain disappears.

The third is splitting the stream into sentences. The IncrementalSentenceChunker class takes text in pieces through feed(), uses a regular expression to separate completed sentences and holds back the tail. At the end, flush() picks up the remainder, which has no final period. The authors are upfront about a limitation: this is not parallel synthesis. The handler is strictly sequential: receive a message, synthesize, send the audio, send synthesis_complete. The only gain is that synthesis of the first sentence starts as soon as that sentence appears. On a live endpoint, they tested three sentences in one session:

• "Hello there." — 73 200 bytes of audio, 1.525 s
• "This is a post deployment check." — 108 000 bytes, 2.250 s
• "It has three sentences." — 98 400 bytes, 2.050 s

SSE with base64 versus binary multiplexing

To deliver audio to the browser, AWS compared two approaches. The first is familiar SSE: audio is encoded in base64 and wrapped in a JSON event. The team measured the cost of this choice on 24 clips (quality, as scored by Amazon Transcribe, did not depend on the encoding):

• Raw PCM: 2 396 400 bytes, 3 195 200 in base64, bitrate 384 kbit/s
• MP3 48k: 308 304 bytes, 411 072 in base64, 49.5 kbit/s
• AAC 32k fMP4: 258 039 bytes, 344 084 in base64, 41.6 kbit/s

Base64 adds almost exactly the expected 33%. There is also a less obvious problem: SSE event boundaries (a blank line) do not line up with MP3 frame boundaries or with how the transport slices data. In one measurement, four logical MP3 frames arrived as six chunks, two of which broke off mid-frame. A consumer that treats every chunk as a complete frame breaks on perfectly healthy traffic. A buffer fixes this: accumulate bytes and process a frame only once the `\n\n` delimiter is found.

The second approach is to drop base64 and SSE entirely on the leg before the model. Binary audio and JSON commands travel over one bidirectional session and are told apart by a single rule: if a chunk parses in full as a JSON object, it is a control frame; otherwise it is audio. Checking the first byte does not work, because PCM can start with 0x7b, which is "{". But a whole PCM chunk will almost never be valid JSON.

The full agent

In the full configuration, a Strands agent with a model from Amazon Bedrock runs in the Bedrock AgentCore runtime and uses three channels: SSE to the browser, a streaming Bedrock call for text, and a bidirectional session with Kokoro for synthesis. The request has a mode field: "bidi" or "batch". In run_bidi, two tasks run in parallel: one reads the Bedrock stream and feeds completed sentences to the chunker, the other reads Kokoro's responses and emits audio.

Both put their results into a shared queue, so audio for early sentences can reach the caller while Bedrock is still writing later ones. run_batch waits for the full text and makes a single synthesis call, which gives an honest baseline for comparison.

On the leg from the agent to the caller, base64 remains anyway: the BedrockAgentCoreApp entrypoint must return JSON-serializable dictionaries, which the SDK itself turns into `data: {json}` frames. A binary frame cannot be passed through. This cost is the same in both modes, so the difference in the measurements is purely the timing of the first audio chunk.

What to choose in practice

If your frontend already uses SSE for text, base64 inside the event is a pragmatic choice: +33% in size and a buffer for reassembly, but the infrastructure is ready. If you are building the voice channel from scratch, or latency is critical, a single binary scheme with JSON control removes both costs and reuses the connection that already carries the text. In either case, test with binary frames specifically, check reassembly across MP3 frame boundaries and do not close the session after every sentence. And before production, update the pinned PyTorch.