Support Agent in Production: Temporal, Sagas, and a Circuit Breaker Protect Refunds from Failures

A customer writes: "My package never arrived, give me my money back." You can write an agent that answers this in an evening: a loop, three tools, and the demo works. Then the process restarts in the middle of a refund, the model provider goes down for half an hour, and the customer hits "send" twice. It turns out the agent is only a small part of the system. The final installment of a Habr series assembles that system step by step, and its main value is that each component appears as a response to a specific breakage.

From a single file to an API, a worker, and Temporal

The initial version keeps the conversation history in process memory. A restart wipes all conversations, customers are served one at a time, and if the process crashes between the refund_payment call and the reply, nobody knows whether the money went back.

Step one: split the script into an API and a worker. The API accepts a message, writes it to the database, and replies to the chat immediately. It does not call the model, so it responds in milliseconds. The agent loop lives in separate workers, which you can add as load grows. The history is now in the database and survives restarts.

But a worker can still crash in the middle of a request. Retrying the refund is dangerous: you may pay twice. Not retrying is bad too: you may not pay at all. This is where Temporal comes in. It writes every step to a journal, and another worker picks up from the same point without re-running completed steps. The agent loop lives in a workflow, while model and tool calls become activities.

Two important clarifications. Temporal has its own database, and customers' business data should not be stored in it: the journal of a single execution is limited to 51 200 events or 50 MB. You run the workers with your code yourself; the Temporal service does not execute them. And you set the Workflow ID: use the conversation ID, and a repeated submission will not create a duplicate. Just do not put personal data in the ID, since it is visible in the UI and in logs.

A retry is not a circuit breaker

When the model provider is down for half an hour, Temporal faithfully retries the activity. A hundred workers and a thousand open requests hammer the dead API in unison, and when it comes back up, the entire queue crashes down on it.

Retries are designed for a failure that clears up in seconds. A long outage calls for a circuit breaker. It counts failures, trips open after a threshold and immediately returns an error without calling the service, then lets a trial call through after some time. The breaker's state must be shared across all workers, otherwise each one will have to learn all over again that the provider is down. That is why it belongs in the LLM gateway between the workers and the providers. LiteLLM, for example, takes a model out of rotation after a series of failures, switches to a fallback, and stores the state in Redis.

The author's practical advice: have the activity distinguish two kinds of errors. A transient one is retried by Temporal's policy. A "breaker is open" error should not be retried: the workflow itself decides whether to switch to a fallback model, wait, or hand the request to an operator. The same breakers go in front of the delivery service and the payment system.

Saga, operator, and router

A refund touches three systems with no shared transaction: the shipping label at the delivery service, the money in the payment system, and the customer notification plus the order update. This is a saga: every step has a compensation. The key concept is the point of no return (pivot transaction). Before the money is refunded, steps can be rolled back; after it, the email to the customer and the order update must be carried through to the end, however many retries it takes. Every action gets an idempotency key, such as the decision ID, so the payment system will not charge the money back a second time.

The model should not process large amounts. The workflow shows the decision in the operator console and waits for a signal. The wait is recorded in the journal and does not occupy a worker, so the operator can respond an hour later.

The last step concerns cost. Not every request needs a full agent loop. A router on a small model sends FAQs to a single call, a "where is my order" question to a predefined chain, and only a disputed refund reaches the agent. The principle is simple: use the lowest level of complexity that reliably solves the task.

What to do with this

You should not build the entire setup at once. The author explicitly advises adding a component when the pain it treats has appeared. If you have one customer at a time and no money at stake, a database is enough. As soon as side effects such as payments appear, you need idempotency and durable execution. As soon as you depend on an external provider, you need a shared breaker.

There are limitations too. The code in the article is pseudocode. The claim that OpenAI's Codex web agent runs on Temporal comes from the vendor itself, and the author found no independent confirmation. But as a map of the typical failures of an agent system, the text is useful to anyone shipping an agent to production. Start with the question of what happens if the process crashes after the riskiest call.