Mentorpiece launched a free course on non-functional testing of AI applications

Mentorpiece has released a free introductory course on non-functional testing of AI applications. The course authors start from a simple idea: for products built on models, answer quality alone is no longer sufficient, because the overall user experience is undermined by cost, latency, instability, and the opacity of the models themselves.

Why AI is More Complex

In classical software development, non-functional checks are often postponed until release or even after the first users. With AI applications, this approach quickly damages the product. Even if a scenario looks functional in a demo, production can reveal completely different limitations: volatile token costs, unstable latency, provider rate limits, empty responses, or quality degradation on real data. For the team, these are no longer secondary details—they become part of the basic check of whether the function can run at all in production.

A separate issue is traceability. An AI model remains a black box even for the team that implemented it: one set of data goes in, an answer comes out, but the logic inside is hidden. The article explains this through the image of the answer "42" from "The Hitchhiker's Guide to the Galaxy": there is a result, but why it is exactly that is unclear. If traceability testing is not established, the product quickly starts returning results that are hard to explain, reproduce, and improve.

Real Cases from Practice

One of the most striking examples in the article is cost testing. A team compared two models for the primary role in an application: popular model A and lesser-known model B. According to test results, model A produced 63% more errors than model B. At the same time, its input tokens cost 75 dollars per million, while model B cost 3.75 dollars. In other words, the cheaper model turned out to be not a compromise, but the better option both in price and quality.

"Model B is 20 times cheaper with much better accuracy."

The second case concerns reliability under load. One AI application used three models from three different providers simultaneously. While several dozen auto-tests ran in parallel, the system behaved normally. But after exceeding a hundred simultaneous tests, failures began: one model started regularly returning a 429 Too Many Requests error, while another returned empty output without an explicit error in approximately 10% of cases. For a user this looks like a random failure, but for QA it is a signal that load and reliability checks are mandatory here.

What's Included in the Course

Mentorpiece's course is designed as an introductory overview for testers who haven't yet worked with AI applications but want to quickly understand where the new risks lie. The material does not attempt to overwhelm the reader with mathematical details of models. Instead, it gathers the main testing areas that most often affect the launch and operation of AI features in a real product.

• cost testing and comparing models by price and error rates
• traceability testing and black box analysis
• reliability, resilience, and load behavior testing
• privacy and data leak testing
• approaches to testing AI agents, RAG, fine-tuned models, data, and LLM-as-a-Judge scenarios

The authors separately raise the practical question of model selection. Their thesis is simple: public benchmarks cannot be blindly trusted, because a real product lives on its own data, with its own constraints on budget, speed, and acceptable error levels.

The course is available for free, and registration is needed only to save progress. In addition to the Mentorpiece platform, it is also posted on Stepik.

What This Means

The topic of AI-QA is rapidly emerging from the status of a narrow specialty. Even teams that don't build their own models already have to test the behavior of external LLMs as part of the product: track costs, catch degradation, monitor failures, and understand why the system answers the way it does. Mentorpiece's free course is an attempt to provide a basic map of this new zone, where non-functional testing becomes not an addition, but a condition for normal operation of an AI service.