Batch API for LLMs: 50 000 Requests at Up to 50% Off, and Three Classes of Errors

When you have a dozen or a hundred LLM requests, the synchronous API behaves well: you send a request and get a response. But once you need to process 50 000 contracts or product listings, you quickly hit TPM and RPM limits, and long-context requests start failing before generation even begins. For jobs like these, OpenAI, Anthropic and Google offer a batch mode, the Batch API. You put all your tasks into one file, send it to the provider and wait up to 24 hours, but you pay less.

How the queue works

Behind the simple word “batch” sit three levels of constraints.

The first level is the batch itself. It is a JSONL file in which each line contains custom_id, method, url and a body with the model and messages. In essence, each line is an ordinary Chat Completions request packed into a single line. You upload the file through the Files API with the purpose “batch”, then create a job. Results do not come back in the order you sent them, so custom_id effectively works as a primary key: you use it to match each response to its source record.

The second level is the enqueued-token pool. The limit is counted not per file but across the total number of tokens in all unfinished batches. If a new batch pushes your organization over the ceiling, the API returns token_limit_exceeded and refuses to create the job. At OpenAI the ceiling depends on your tier: the higher the tier, the more tokens you can keep in the queue.

The third level is the completion window. The 24 hours is an SLA, not a forecast. In practice, batches most often finish within a few hours, and Anthropic says most of its batches are done in under an hour. But you should plan for the worst case. The completion_window parameter currently accepts only the value “24h”.

Where the pipeline breaks

The author of the original article splits errors into four classes, and the traps here are not trivial.

Before the batch is created. A stray comma in the JSONL, a duplicate or missing custom_id, a wrong endpoint. According to the author, Anthropic validates request parameters only after the entire batch has finished: you can submit 100 thousand rows, wait an hour and then find out that three of them have a malformed max_tokens. So validate the file locally before uploading: parse every line, check that each custom_id is unique and that model and messages are present. Pydantic models work well for this.

At the queue level. The main error here is token_limit_exceeded. The fix is to count tokens before sending and to keep your own queue on the application side. If the budget doesn't fit, wait for earlier batches to finish or retry with exponential backoff: 30, 60, 120 seconds and so on.

Inside the batch. This is the trickiest category. The batch is completed, but in the output some rows are succeeded, some errored and some expired. If your code simply reads content and writes it to the database, the problem rows are silently lost. At Anthropic, the errored, canceled and expired counters in request_counts stay at zero until the whole batch finishes, so you cannot gauge the scale of a failure in advance. A sensible scheme uses three buckets: invalid_request rows get fixed and resubmitted, server errors are retried with a delay, and expired rows go into a new batch because their window has already closed.

When polling for status. If you hit the status endpoint every five seconds, you can get a 429 on the polling itself. A naive loop will read that as “batch not ready”, when in fact you have hit a rate limit on reads. For batches of up to a thousand rows, poll every 30 seconds; for larger ones, every 2–5 minutes. If a batch hangs for more than a few hours, check whether it is stuck in the validating stage.

What it really costs

The 50% discount is a ceiling, not a guarantee. The final figure depends on how many requests ended up expired, how many had to be resubmitted and whether system-prompt caching was used. In a good scenario the savings are even higher, because cached input is cheaper too. In a bad one, you pay twice for the same rows. One more important detail: at OpenAI, when you cancel a batch, unfinished requests are canceled, but you are charged for those already completed.

It is also worth noting that the original is partly written in a promotional vein: the author pushes Bothub, an OpenAI-compatible gateway that accepts Russian cards and claims 45% savings. The idea of a compatible endpoint is sound, but you should check the terms and reliability of any intermediary yourself rather than relying on a referral link.

What to do in practice

A batch is a different contract: you get a discount and a separate limit pool, but you take on the work of parsing results. If you are building bulk document processing, classification or labeling, build a few things into your pipeline:

• local JSONL validation before sending;
• token counting and a semaphore on the enqueued limit;
• parsing the output by result type and resubmitting only the failed rows;
• polling with backoff and logging of created_at.

And the main rule: don't resubmit an entire batch because of three failed rows, since you have already paid for the successful ones.