Open-source LLM gateway KTMS 1.0 plugs your own GPUs into the cloud and survives outages

KTMS (Keystone. The Model Switchboard) is an open-source LLM gateway written in Go: a single point through which a company's applications reach models from different providers. The authors have released version 1.0 and written up how the gateway behaves in production. You can connect your own GPUs and send requests to them before the paid cloud. For each of the three dependencies (Postgres, Redis, the accounting service), the failure behavior is defined in advance. Encryption keys rotate without loss, and releases are signed and verifiable. Updates run without stopping the service.

Your own GPUs first, the cloud as backup

Many teams have a Mac Studio or a Linux machine with a good card that sits idle half the time. Meanwhile the cloud bill keeps growing for requests that a local model could handle. KTMS turns such machines into a separate vendor named local.

An administrator issues a node key over gRPC (it is shown only once) and registers a model. On the machine itself, a single script does the rest. It installs Ollama, the model and the node agent, sets up background services and keeps the computer from going to sleep. On macOS, no administrator rights are needed. If something breaks, the same script with the --doctor flag reports the cause: Ollama is not responding, the model is not loaded, Ollama is running on the CPU through Rosetta, or a wrong --tls is set. Ollama, vLLM, TGI and llama.cpp are supported.

The gateway connects to the node itself, so the node has to be reachable. The agent can set up a Cloudflare tunnel and a filter in front of the engine. Only POST /v1/chat/completions and GET /v1/models pass through it, so a model cannot be downloaded or deleted from outside.

Node selection works in three steps:

• Filter. Nodes that have the requested model, a free slot and a sufficient context window stay in. Healthy nodes rank ahead of suspect ones, and "warm" nodes (model already in memory) rank ahead of "cold" ones.
• Weight. The base weight is multiplied by a load penalty, which grows quadratically with slot occupancy, and by a speed adjustment. The adjustment is computed from generation speed (tokens per second, weight 0.7) and time to first token (weight 0.3), relative to the median across all nodes serving that model.
• Pick. Smooth weighted round-robin is applied, as in nginx.

The main protection against slowdowns is a queue estimate based on Little's law: queue depth multiplied by the expected request duration. If the result exceeds the model's queue_timeout_ms (2000 ms in the example), the request goes straight to the next model in the chain. For interactive clients, the limit is 250 ms. A slot is a lease in Redis that is renewed while generation is running.

Placement policies are nodes_first (the default), nodes_only (if no node is free, the result is an error and the cloud is not touched), cloud_first and cloud_only. You can choose one with a header, a request field or a model suffix, for example auto:nodes-only:high.

What the gateway does when Redis and Postgres go down

The principle: a failure of accounting must not turn into a failure of the service.

• Redis. Request and token limits are skipped without being counted. For dollar limits, you choose a mode: open lets requests through, closed rejects them. The cache counts as a miss. Node state is unknown, so nodes are skipped and the cloud answers. After three network errors in a row, the gateway stops calling Redis entirely for 2 seconds, so a hung Redis costs a request only one timeout.
• Postgres. Keys verified within the last 5 minutes keep working, and the rest get auth_unavailable. Verification is capped at 3 seconds. Configuration comes from the last successful snapshot. A batch whose idempotency cannot be verified is rejected so that it is not paid for twice.
• Accounting service. Events accumulate in memory (up to 10 000), and requests do not wait for them.

During Postgres failures, /readyz returns 200 with a degraded status while the known-keys window is active, and 503 after it. Kubernetes will not restart pods prematurely. Five failures in a row open the vendor's circuit breaker, and after 30 seconds one trial attempt goes through. It is worth setting an alert on ktms_dependency_errors_total.

Keys, release signatures and zero-downtime updates

Vendor keys are stored in Postgres, encrypted with AES-256-GCM. The keyring with the encryption keys is passed in the KTMS_ENCRYPTION_KEYS variable and is not stored in the database. Rotation takes three steps. First, add the new keyring version to all replicas and wait for the rollout to finish. Then call RotateEncryption: in a single transaction, vendor keys and node tokens are re-encrypted. Finally, remove the old version once no backups remain that require it. A dump together with the keyring yields all keys in plaintext, so the two must be stored separately.

Images since version 0.14.0 are signed with cosign in keyless mode: the signature is bound to the identity of the GitHub Actions workflow, so there is no key to steal. Images ship with an SBOM (a list of components) and SLSA provenance (a record of where and how the image was built). Every pull request is checked with govulncheck, and a release does not ship without buf breaking and an upgrade test. The authors state a caveat openly: the signature confirms provenance, not safety. Anyone who gains rights to the release workflow can sign a malicious image too.

Within 1.x, the public API does not break. Migrations are applied at startup under an advisory lock and only add schema, so old replicas keep running on the new database while new ones come up. There is no rollback: to go back, you have to restore from the backup taken before the update. On SIGTERM, requests get 30 seconds, and the node agent goes into drain first.

For teams with idle GPUs, this is a ready-made way to cut the cloud bill without the risk of being left without an answer under overload. Before launch, check TLS (the gateway itself speaks plaintext gRPC), remove BOOTSTRAP_ADMIN_KEY, create a separate key per client and watch Postgres connections (up to 25 per replica).