AWS shows how to run llama.cpp on SageMaker and get probabilities over a set of options

The AWS Machine Learning Blog has published a step-by-step guide to deploying decision models on Amazon SageMaker AI with the llama.cpp engine. The setup has two layers. First, you stand up an endpoint based on the official AWS Deep Learning Container for llama.cpp, which serves OpenAI-compatible chat completions. Then you extend the runtime with the SemIf library, so the model returns probabilities over predefined options instead of generated text. All of this works on both CPU and GPU.

The material is hands-on: it is a tutorial, not the announcement of a new product. But it shows how AWS suggests handling one of the most common enterprise scenarios: classification and decision-making on small models, without heavy GPU infrastructure.

What the stack is made of

llama.cpp is an open-source inference engine in C/C++ that can run quantized models in GGUF format and does not require a GPU. That is why it is popular wherever teams need cheap, predictable endpoints. AWS packaged it into its own Deep Learning Container, so developers do not have to build the image by hand or sort out dependencies.

Key elements of the solution, according to AWS:

• Platform: Amazon SageMaker AI, managed inference endpoints.
• Runtime: AWS Deep Learning Container for llama.cpp.
• Interface: OpenAI-compatible chat completions, meaning existing clients and SDKs connect without being rewritten.
• Extension: SemIf, which returns probabilities over declared options.
• Hardware: CPU or GPU, with the choice left to the developer.

OpenAI-format compatibility solves a practical migration problem. If an application already calls a chat API, switching to your own endpoint comes down to changing the address and the key.

Why probabilities instead of text

A decision model answers a question with a finite set of outcomes: approve or reject an application, send a ticket to billing or to tech support, flag a message as a violation or let it through. A regular LLM returns a string that has to be parsed. The response format drifts, the model adds explanations, and sometimes it invents an option that is not on the list.

By its description, SemIf addresses this at the runtime level. The developer declares the allowed options and gets back a probability distribution over them. You can pick the top option, but the more interesting part is something else: a confidence threshold becomes possible. If the best option scores 0.92, the action runs automatically. If it scores 0.48, the case goes to a human. This turns a language model into a component with measurable calibration that can be built into an ordinary business process.

For engineers, it is also a testing question. Probabilities can be logged, compared across model versions, and used to build metrics, which is not possible with arbitrary text.

What the announcement does not say

The short summary contains no benchmarks, prices, list of supported models, or details of how SemIf is implemented. It is unclear exactly how the probabilities are calculated: from the logits of the options' first tokens or some other way. That determines how far they can be trusted on long or compound options. There is also no CPU-versus-GPU comparison of latency and cost per request. Teams will have to gather that data themselves.

Where to start

A practical sequence for a team that wants to test the approach:

• Pick one narrow task with 3–10 outcomes, for example routing support requests.
• Take a small quantized model in GGUF and run it on a CPU instance. This is the cheapest way to start.
• Assemble a labeled sample and measure how closely SemIf probabilities track real accuracy. Tune thresholds for automatic decisions and for escalation.
• Only then compare against a GPU instance on latency and price per thousand requests.

If calibration turns out to be poor, it is better to find out before production. If it is good, the team gets a managed endpoint with a familiar API and clear economics.