Ollama and LiteLLM: Turning a Python Script into a Full Console LLM Chat

In the second part of the guide to Python, Ollama and LiteLLM, the author moves from a one-off model call to a full console chat. The simple script becomes a starting point for a local AI application, with a persistent dialogue, a system role and protection against typical failures.

From script to chat

The first step here is not changing the model but changing the usage scenario. A one-shot request is fine for checking that the Python, Ollama and LiteLLM stack responds at all. But as soon as you want to ask a follow-up, rephrase something or continue a thought, such a script quickly hits a ceiling.

A console chat solves this in the most direct way: the program does not exit after one answer. It stays in the dialogue and lets you talk to the model like an ordinary conversation partner in the terminal.

For a developer, this is an important turning point. Instead of "call the API, get text back", a minimal application interface appears: user input, message history, the model's reply and the control logic around them. LiteLLM is convenient here as a single layer for calling models, while Ollama handles local execution. As a result, even a small learning project starts to resemble a real product that you can already run, test and gradually extend without rebuilding the foundation.

What appears in the code

The next layer is the details that turn a chat from a plain input/output loop into a managed program. In examples like this, the valuable parts are not "magic" features but basic engineering elements: who sets the model's role, where the context is stored, and what happens if a request fails. These are what make it feel like a working foundation for a local assistant rather than a five-minute demo.

• A conversation loop that accepts new messages until an exit command
• A system prompt that sets the model's role, tone and behavioral boundaries
• A message list, so the model sees previous turns and keeps context
• Basic input validation, so empty requests are not sent
• Exception handling, so the program does not crash after the first failure

Each of these items looks simple, but together they change the quality of the interaction. The system prompt is not just for the bot's "personality": it is a convenient place to fix response rules, format, language and constraints. Message history allows a coherent conversation instead of re-explaining the task on every turn. Error handling saves debugging time: if the local model hangs, Ollama is not running or LiteLLM raises an exception, the session is not lost entirely.

First steps toward an application

A separate point concerns the first steps toward a "live" AI application. A console interface looks modest, but it is the easiest place to check how a bot behaves in a real dialogue, where the user phrases thoughts imperfectly, asks follow-ups and keeps changing context.

Weak spots show up quickly here: a system prompt that is too generic, an awkward output format, no commands for exiting or restarting, and unclear errors when loading the model.

Such a skeleton is easy to extend without extra architecture. On top of it you can add token streaming, separate commands such as /clear to reset history, model switching, dialogue logging or tool integration. But the value of the current step lies elsewhere: the author shows that a useful interface starts not with a GUI or a web app but with a reliable conversation loop in the terminal. If this layer is built carefully, it is easier to grow later toward both a product and further experiments.

What this means

For anyone building local AI tools in Python, this stage is mandatory. The Ollama and LiteLLM combination becomes interesting not at the moment of the first successful reply but when a proper conversation framework appears around the model. A console chat is the minimal form of that framework: simple enough to start with and useful enough to build the next layer of functionality on.