MIT Technology Review: Small language models become the foundation for AI implementation in the government sector

Government agencies can no longer afford to discuss AI at the level of pilots alone: for technology to reach actual work, they need not the largest models, but those that can be embedded in a rigid security contour, local infrastructure, and responsibility frameworks. This is precisely why interest is increasingly shifting toward small language models for departments and public institutions — specialized systems that are simpler to control, cheaper to run, and easier to verify against compliance requirements. In business, there's a common logic of "plugged in a cloud LLM and let's test scenarios," but for government organizations it often doesn't work.

There are higher requirements for data protection, stricter rules for information movement, greater importance placed on fault tolerance and solution auditability. In many cases, you can't rely on constant internet access, centralized clouds, or free data exchange between systems. This is why, despite strong pressure to accelerate implementation, many AI projects in the public sector get stuck between demonstration and industrial launch.

These barriers are confirmed by numbers. According to Capgemini research, 79% of leaders in the public sector cite data security as the main obstacle, 74% — lack of trust in AI answer quality, and 71% — questions of data sovereignty and localization. At the same time, interest in the technology is high: 64% of organizations are already studying or conducting initiatives in generative AI, but only 21% have reached pilots or actual implementation.

The gap between the desire to use AI and the ability to safely introduce it into the work cycle remains the main bottleneck. Against this backdrop, SLMs look like a more realistic compromise. Such models can be adapted to a specific agency, department, or set of tasks, rather than trying to overlay a universal LLM with the entire array of restrictions.

They require fewer computational resources, often can work locally or in an isolated environment, and allow you to keep sensitive data outside the model itself, feeding it on request through search and retrieval mechanisms. For environments with limited internet and a modest GPU fleet, this is not a convenience but a basic condition for implementation. This is especially important where data cannot be taken outside the perimeter, and every answer must be explainable and tied to a verifiable source.

In practice, this means a combination of a small model, corporate search, and strict access rules. In such a scheme, the system doesn't "guess" the answer from general knowledge, but pulls relevant documents, PDF fragments, tables, images, or archive materials, ranks them, and only then formulates the answer. For government agencies, this provides more useful scenarios: searching regulatory documents, processing citizen requests, summarizing files and cases, supporting analysts and front-office staff.

What matters is not a benchmark record, but the ability to log actions, restrict access rights, reduce hallucinations, and reproduce the answer logic when checking. In this approach, the key question shifts from "which model is smarter" to "which architecture is more reliable." The public sector needs not just a chatbot, but a full-fledged operational layer for AI: with test contours, decision logging, risk management, unified security policies, and compatibility between contractors and internal systems.

The higher the stakes — from social services to defense and investigations — the more important it is that AI is not only useful but manageable at every stage: from data request to final answer. Without such infrastructure, even a powerful model remains an impressive but poorly controlled demonstration. This means that for the public sector, the next wave of AI implementation will likely be built not around the most well-known universal models, but around specialized, locally controlled and verifiable systems.

If this scenario works out, it may be state and public organizations that show the market how to turn generative AI from an impressive demo into an infrastructure tool with clear responsibility, control, and real utility.