MiniJev Reads Logits and Speeds Up Query Classification by 5.4x

If a model needs to select a support department, assign an urgency score, or answer yes or no, it does not need to produce a full text response. Yet a conventional LLM pipeline still forces it to generate JSON sequentially: braces, keys, quotation marks, and values. Every new token waits for the previous one, even though the decision itself is often already visible in the probability distribution after the prompt has been processed.

The author of the MiniJev prototype proposes reading that decision directly from the logits—the numbers the model computes before selecting the next token. The approach resembles the Jev interface from TypeSafe AI but does not attempt to reproduce the model itself: the experiment uses a standard Qwen 2.5 1.5B Instruct model without fine-tuning. Everything runs on a free Nvidia T4 in Google Colab.

How to Get a Decision Without Generating Text

Consider a request: “I was charged twice; refund one payment.” The system asks the model three questions: whether the customer is angry, which department should handle the request, and how urgent it is. For each question, the options are listed in advance and labeled A, B, C, and so on.

After a single forward pass, the model already has logits for the possible first response token. MiniJev keeps only the logits for valid letters, applies softmax, and obtains a probability distribution. For the department, it might look like this: billing — 99.99%, technical support — 0.005%, sales — 0.001%. No braces or field names need to be generated.

The code accounts for two practical nuances. First, the tokenizer may encode a letter differently with and without a leading space, so the probabilities of valid single-token variants are combined. Second, the model sometimes favors the first or last item regardless of meaning. To counter this bias, the options are cyclically reordered and their log probabilities are then averaged. The trade-off is additional prompts and computation, especially when there are many options.

Fast, but Honest Probabilities Take More Work

On one test request, MiniJev's median runtime was 0.1959 seconds. Generating equivalent JSON took 1.0514 seconds—about 5.4 times longer. This is a useful reference point, but not a universal result: the measurement used one model, one GPU, and one type of query. The comparison also depends on batching, JSON length, and the number of option permutations.

Raw probabilities cannot automatically be interpreted as genuine confidence. The model was trained to predict the next token, not to report how often it is correct. The author tested calibration on 600 SST-2 examples: 200 were used to fit Platt scaling and 400 for testing. Accuracy remained at 92.7%, while expected calibration error fell from 6.8% to 2.5%. The result is encouraging, but it covers only one dataset and one split.

Where to Use It

The approach is suitable for request routing, moderation flags, intent detection, selecting an agent's next action, and fixed-scale scores. It guarantees that the response remains within the specified list, but it does not guarantee truth: the model can confidently choose the wrong option.

The practical takeaway is simple. If the answer set is known in advance, first test reading logits instead of generating structured text. Then collect at least several hundred labeled examples from the target environment, calibrate the probabilities, and measure not only latency but also accuracy, ECE, and the cost of additional permutations. This technique is not suitable for open-ended answers or complex explanations, where generation is still necessary.