Flag Soft: Dali Trial benchmark helped select LLMs by quality, speed, and cost

Choosing an LLM for a real product rarely comes down to comparing beautiful demos. The author approached the task from a practical angle: while searching for a model for his first pet project, he assembled his own benchmark called "The Dali Trial" and tested popular LLMs not by their grand promises, but by three things that actually matter for implementation — answer quality, speed, and cost. The idea grew from a quite everyday engineering problem.

When you need to choose a model for your own project, the abstract question "which LLM is best" quickly turns into a set of practical constraints. One model writes convincingly but answers too slowly. Another fits the budget but loses the thread in long instructions.

A third consistently passes tests, but the final cost makes it unsuitable for a mass-market product. It was at exactly this point that the homemade test appeared, which turned out to be useful not only for a personal experiment, but also for the product solutions of Flag Soft. The "Dali Trial" is based on simple but sound logic.

If a model is planned to be embedded in a product, it should be compared not by a single impression from a chat, but by the same set of tasks. Quality in such an approach means not just "like the answer or not," but the model's ability to preserve meaning, follow instructions, not lose details, and deliver a result that can be used without lengthy manual editing. Speed is no less important: for an internal tool, you can tolerate extra seconds, but in a user-facing service, every delay hits retention and conversion.

Cost is the third mandatory parameter, because even a powerful model can turn out to be too expensive when scaling to thousands of requests. This is the value of the benchmark: it doesn't seek an absolute champion, but shows the balance. In practice, the model that simply writes better almost never wins.

The one that wins is the one that delivers acceptable quality in the right time and at a price compatible with the product's unit economics. For a company wanting to embed an LLM in a real service, this is far more useful than impressive tables with abstract scores. This evaluation method helps you see in advance where the bottleneck will appear: in response delay, in token budget, or in unstable model behavior on similar queries.

Separately interesting is the author's applied conclusion: the benchmark helped select not "the smartest" model in general, but the optimal LLM for integration into Flag Soft's products. This is an important distinction. Teams often start implementation with a top-tier model, then are forced to roll back to a cheaper or faster alternative.

Here the logic is reversed: first real requirements are formulated, then a model is selected to meet them. This order reduces the risk of expensive reworks, when the architecture is already tied to a provider that doesn't deliver the economics, response speed, or expected service level. The author's approach is useful also because it reflects the real state of the LLM market.

For different scenarios, different models may win: text generation, summarization, knowledge search, operator assistance, autocomplete in the interface, or processing customer requests. The same candidate can perform excellently in creative tasks and fail where strict instruction-following discipline is needed. That's why custom benchmarks become not a luxury but basic hygiene for any team planning to pay for a model from its own budget and be responsible for user experience.

The main point of "The Dali Trial" is simple: LLMs should be chosen the same way as any infrastructure technology — through verifiable metrics, not through hype. If a team has its own set of tasks, a time-response limit, and a clear budget, it will almost certainly get a more accurate answer than from a general leaderboard. For the market this is another signal: the era of choosing a model "by reputation" is ending, and engineering pragmatism takes center stage.