5 small open models with tool calling: agents that don't need the cloud
Small language models can now call functions and drive tools, which is key to decentralized AI agents. Instead of relying on cloud services, you can now run compact models locally with full support for tool calling and structured calls. All five models on this list are open, compact, and ready to deploy on servers and devices without an internet connection.
AI-processed from KDnuggets; edited by Hamidun News
For a long time, small language models couldn't compete with cloud services on one key skill: driving tools through tool calling. That is now changing. A new generation of compact, open models not only supports structured function calls but also stays light enough for local deployment.
What is tool calling and why does it work?
Tool calling is a model's ability to invoke external functions, scripts, or APIs directly, instead of just writing code in its reply. The model sees a list of available functions with descriptions, their parameters, and data types, and decides on its own which function to call and with which arguments. This is critical for AGI agents: they can manage databases, download files, send emails, and book meetings, all without direct human involvement.
Structured output (JSON-shaped responses) guarantees that the model returns its result in a correct, predictable format that a program can parse and use downstream. Until recently, only large models (GPT-4, Claude 3) could do this reliably. Now small models have also learned to emit structured JSON without glitches.
Why are small models now competitive?
Small models (7B-13B parameters) have several advantages over large ones. They are cheaper to develop and run inference on, private by default (they don't send data to the cloud), and faster to respond. They don't require cloud services or powerful enterprise hardware: a mid-range GPU or even a decent CPU is enough.
Add tool calling support to a small model like this, and you get a fully functional AI agent that you can run on your own server, laptop, or even smartphone without internet access. This opens the door to private enterprise agents with guaranteed data confidentiality. A company can run an agent inside its secure network without sending a single request to the cloud.
On top of that comes licensing flexibility: all of these models are open, and you can use them for commercial purposes without asking for permission.
5 models ready to use
Here are five small models that already support full tool calling today:
- Llama 3.1 (Meta) — the base 8B version, with good documentation and tool-calling examples; the most proven and stable on the list
- Mistral 7B — compact and very fast, with a good balance of quality and size; popular in enterprise environments
- PhiLM 3 (Microsoft) — optimized specifically for structured output and engineering tasks; minimal memory requirements
- OpenChat 3.5 — focused on functions and tool control; strong benchmarks in tool-calling tests
- Neural Hermes 2.5 (finetuned Mistral) — handles complex multi-step call chains and errors better than the rest
All five can be downloaded from Hugging Face in a few minutes and run locally without internet access. Inference time (the response to a request) ranges from 50 to 200 milliseconds on a modern GPU or fast CPU.
What does this mean for the industry?
The era of the cloud monopoly on AI agents is over. Now even small startups and corporations can build private, fully functional AI agents that are no slower and no dumber than cloud counterparts such as the OpenAI API or Claude via cloud.
This means the entire AI infrastructure is gradually moving from the cloud to on-premise. In the coming months, we expect a surge of tools and frameworks for local agent deployment (along the lines of LM Studio and Ollama, but with proper tool calling support).
For developers, this opens up an entirely new market: private AI agents for large corporations, government agencies, healthcare, and fintech. Anywhere the cloud is off-limits for political or legal reasons, local models are the only way forward.
*Meta is designated an extremist organization and banned in the Russian Federation.
- Nov 18, 2026Opus 5.5 Tells 19 Safety Tests From 36 Ordinary Tasks Without a Single Error
- Nov 18, 2026GPT 6.2 Sol Ultrafast Delivers 150 Tokens/s but Burns Through Limits 6x Faster
- Nov 18, 2026StarSling Cuts Mastra's Slowest Build from 29 Minutes to 5
- Nov 18, 2026AI Agents Shipped 228 Game Releases Without Human Review; Tests Sent Back 25% of Tasks
Key Questions Answered
What is tool calling and why does it work?
Tool calling is a model's ability to invoke external functions, scripts, or APIs directly, instead of just writing code in its reply. The model sees a list of available functions with descriptions, their parameters, and data types, and decides on its own which function to call and with which arguments. This is critical for AGI agents: they can manage databases, download files, send emails, and book meetings, all without direct human involvement.
Why are small models now competitive?
Small models (7B-13B parameters) have several advantages over large ones. They are cheaper to develop and run inference on, private by default (they don't send data to the cloud), and faster to respond. They don't require cloud services or powerful enterprise hardware: a mid-range GPU or even a decent CPU is enough. Add tool calling support to a small model like this, and you get a fully functional AI agent that you can run on your own server, laptop, or even smartphone without internet access.
What does this mean for the industry?
The era of the cloud monopoly on AI agents is over. Now even small startups and corporations can build private, fully functional AI agents that are no slower and no dumber than cloud counterparts such as the OpenAI API or Claude via cloud. This means the entire AI infrastructure is gradually moving from the cloud to on-premise. In the coming months, we expect a surge of tools and frameworks for local agent deployment (along the lines of LM Studio and Ollama, but with proper tool calling support).
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.