Keystone plugin teaches Claude Code to set up an LLM gateway: 8 skills, keys kept out of chat

Ask an agent to "set up an LLM gateway for us" and it will confidently invent environment variables that don't exist, and you will spend an hour figuring out why the config doesn't work. An LLM gateway is a single point through which all your services reach OpenAI, Anthropic, Gemini and other providers: it stores the keys, picks the model and fails over when a provider goes down. The authors of Keystone (KTMS), a self-hosted gateway written in Go, solved the problem with engineering rather than prompt-level persuasion.

They released a Claude Code plugin with 8 skills, which are ready-made instructions that know the real commands, flags and config fields, and an MCP server through which the agent sees what is actually running. Keys do not end up in the chat.

What the 8 skills do

Installation takes two commands: add the ai-process/ktms marketplace and install the ktms@ai-process plugin. You separately need the ktms CLI in your PATH (from a release archive or via go install). The plugin version matches the Keystone version.

The skills cover the whole path: assess (evaluates the system and gives a verdict on whether you need a gateway), deploy (Compose or Helm, secrets, first-launch order), integrate (moves one service's LLM calls to the gateway), configure (routing, keys and limits in ktms.yaml, with a change plan), migrate (moving from another gateway or from direct SDK calls), node (a local GPU node with cloud fallback), usage-reports (spending, reports, alerts) and production-review (checks before and after launch). You don't need to remember the names: the agent picks the right skill itself, and a slash command runs one explicitly.

To keep the skills from going stale, Keystone's CI reads every SKILL.md and fails if it mentions a command, flag, MCP tool or README section that doesn't exist in that version. This removes a common problem: an instruction names a flag that was removed two releases ago, and the agent starts improvising.

What it looks like on a demo project

The example has three services: a support bot on the OpenAI SDK, a search service on the Anthropic SDK with OpenAI embeddings, and a nightly worker that calls Gemini about 40 000 times per night. The assess skill changes nothing. It finds all five LLM calls with file and line, and it flags things nobody asked about. All the services load a shared env_file, so each of them gets all three vendor keys. The gateway will ignore temperature=0.3 in one of the calls, so answer quality needs to be checked after the switch. The skill also noticed that rerank.ts doesn't rerank anything and only explains the results.

The verdict is an honest one: the gateway is justified, but if the system had only the support bot on gpt-4o, it would not be needed. The plan starts with the nightly worker: that is where most of the money and risk sit, users don't see the call, and a failed night can simply be rerun.

The integrate skill puts the address, key, timeouts, retries and cost attribution into one small client module. It replaces a hardcoded model with a complexity tier plus a one-line justification: low for classification and short answers, medium for ordinary generation, high for reasoning and long documents. It disables SDK retries and implements the error contract once: wait for Retry-After on 429, back off on 503, and do not retry configuration errors. The final check runs against the real routing: the ktms route explain command (output from Keystone 1.3.6) shows the model chain. If it ends with the status kept, everything is fine; no_vendor_key means a key is missing, and you can see that before the service is rolled out.

Where the trust boundary lies

The agent deliberately leaves the ktms init command to the human. In your terminal, without echo, it asks for the admin key and vendor keys, writes ktms.yaml with default models and rules, shows the plan and applies it.

From there, configure works like Terraform: an edit, ktms config plan, the plan shown to the human, and application only after approval. The restrictions are built into the tools, not the prompt:

• The MCP server is read-only by default; write tools appear with the --allow-write flag;
• config_plan returns a plan_id, and config_apply applies exactly that plan once and will refuse if the configuration has changed in the meantime;
• the vendor key is taken only from KTMS_VENDOR_KEY_* variables, so the agent cannot slip the admin key or another secret into the config;
• names and values coming from Keystone are marked as data, not instructions.

Limitations and practical takeaways

This is a plugin from the gateway's own authors, and it works only with Keystone. The gateway's OpenAI-compatible API ignores temperature, top_p, stop and seed, and embeddings need the exact model id, so some calls don't carry over one to one. The examples come from a demo project, not from someone else's production system.

The lesson is broader than this one gateway. If you want an agent to configure infrastructure, give it skills that are checked in CI against the product version, MCP that is read-only by default, and a plan with confirmation instead of direct writes. Leave the secrets to the human. The skills are ordinary Agent Skills, meaning folders with a SKILL.md, so you can copy them into Codex, Cursor or another agent that reads them, and connect the two MCP servers. For assistants without skills and MCP, the same instructions are available as prompts.