Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)
ReleaseInsight · On RadarKDnuggets→ original

5 small open models with tool calling: agents that don't need the cloud

Small language models can now call functions and drive tools, which is key to decentralized AI agents. Instead of relying on cloud services, you can now run compact models locally with full support for tool calling and structured calls. All five models on this list are open, compact, and ready to deploy on servers and devices without an internet connection.

AI-processed from KDnuggets; edited by Hamidun News
5 small open models with tool calling: agents that don't need the cloud
Source: KDnuggets. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceKDnuggets ↗
Editor-in-Chief
Key metric #1Llama 3.1 (Meta) — the base 8B version, with good documentation and tool-calling examples; the most proven and stable on the list
Key metric #2Mistral 7B — compact and very fast, with a good balance of quality and size; popular in enterprise environments
Key metric #3PhiLM 3 (Microsoft) — optimized specifically for structured output and engineering tasks; minimal memory requirements

For a long time, small language models couldn't compete with cloud services on one key skill: driving tools through tool calling. That is now changing. A new generation of compact, open models not only supports structured function calls but also stays light enough for local deployment.

What is tool calling and why does it work?

Tool calling is a model's ability to invoke external functions, scripts, or APIs directly, instead of just writing code in its reply. The model sees a list of available functions with descriptions, their parameters, and data types, and decides on its own which function to call and with which arguments. This is critical for AGI agents: they can manage databases, download files, send emails, and book meetings, all without direct human involvement.

Structured output (JSON-shaped responses) guarantees that the model returns its result in a correct, predictable format that a program can parse and use downstream. Until recently, only large models (GPT-4, Claude 3) could do this reliably. Now small models have also learned to emit structured JSON without glitches.

Why are small models now competitive?

Small models (7B-13B parameters) have several advantages over large ones. They are cheaper to develop and run inference on, private by default (they don't send data to the cloud), and faster to respond. They don't require cloud services or powerful enterprise hardware: a mid-range GPU or even a decent CPU is enough.

Add tool calling support to a small model like this, and you get a fully functional AI agent that you can run on your own server, laptop, or even smartphone without internet access. This opens the door to private enterprise agents with guaranteed data confidentiality. A company can run an agent inside its secure network without sending a single request to the cloud.

On top of that comes licensing flexibility: all of these models are open, and you can use them for commercial purposes without asking for permission.

5 models ready to use

Here are five small models that already support full tool calling today:

  • Llama 3.1 (Meta) — the base 8B version, with good documentation and tool-calling examples; the most proven and stable on the list
  • Mistral 7B — compact and very fast, with a good balance of quality and size; popular in enterprise environments
  • PhiLM 3 (Microsoft) — optimized specifically for structured output and engineering tasks; minimal memory requirements
  • OpenChat 3.5 — focused on functions and tool control; strong benchmarks in tool-calling tests
  • Neural Hermes 2.5 (finetuned Mistral) — handles complex multi-step call chains and errors better than the rest

All five can be downloaded from Hugging Face in a few minutes and run locally without internet access. Inference time (the response to a request) ranges from 50 to 200 milliseconds on a modern GPU or fast CPU.

What does this mean for the industry?

The era of the cloud monopoly on AI agents is over. Now even small startups and corporations can build private, fully functional AI agents that are no slower and no dumber than cloud counterparts such as the OpenAI API or Claude via cloud.

This means the entire AI infrastructure is gradually moving from the cloud to on-premise. In the coming months, we expect a surge of tools and frameworks for local agent deployment (along the lines of LM Studio and Ollama, but with proper tool calling support).

For developers, this opens up an entirely new market: private AI agents for large corporations, government agencies, healthcare, and fintech. Anywhere the cloud is off-limits for political or legal reasons, local models are the only way forward.

*Meta is designated an extremist organization and banned in the Russian Federation.

Story

Key Questions Answered

What is tool calling and why does it work?

Tool calling is a model's ability to invoke external functions, scripts, or APIs directly, instead of just writing code in its reply. The model sees a list of available functions with descriptions, their parameters, and data types, and decides on its own which function to call and with which arguments. This is critical for AGI agents: they can manage databases, download files, send emails, and book meetings, all without direct human involvement.

Why are small models now competitive?

Small models (7B-13B parameters) have several advantages over large ones. They are cheaper to develop and run inference on, private by default (they don't send data to the cloud), and faster to respond. They don't require cloud services or powerful enterprise hardware: a mid-range GPU or even a decent CPU is enough. Add tool calling support to a small model like this, and you get a fully functional AI agent that you can run on your own server, laptop, or even smartphone without internet access.

What does this mean for the industry?

The era of the cloud monopoly on AI agents is over. Now even small startups and corporations can build private, fully functional AI agents that are no slower and no dumber than cloud counterparts such as the OpenAI API or Claude via cloud. This means the entire AI infrastructure is gradually moving from the cloud to on-premise. In the coming months, we expect a surge of tools and frameworks for local agent deployment (along the lines of LM Studio and Ollama, but with proper tool calling support).

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…