Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseCode / SDK · ActionableHabr AI→ original

Developer builds an email AI agent that handles text, photos and voice through a single API key

For an AI agent to actually help with email, it needs three models: one to read messages and photos, one to recognize speech, and one to voice the replies. Each usually comes with its own account and billing. The author of AI Inbox, built on FastAPI and the OpenAI Agents SDK, connected all three through a single key: you can photograph an invoice, and the agent creates a task with a due date from it. To reuse the setup in your own projects, it is enough to change base_url and the model identifiers in .env.

AI-processed from Habr AI; edited by Hamidun News
Developer builds an email AI agent that handles text, photos and voice through a single API key
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Three rolesan LLM for text and photos, ASR for speech, TTS for voice output
Test curl request154 input and 125 output tokens, 279 in total
Key metric #3An email is sent only after a separate confirmation of the draft

Imagine you want to build an AI assistant for email that does more than chat: it reads messages, prepares summaries and creates tasks. Add photos and voice, and you quickly find that you need three different models: one for text, images and tool calling, a second for speech recognition (ASR), and a third for text-to-speech (TTS). Each usually has its own access, its own billing and its own settings.

The author of an article on Habr showed how to build such an application, AI Inbox, and connect all three models through one provider, with one key and one balance. For example, you upload a photo of an invoice, ask the agent to move its data into a task, and the agent does it on its own. For a developer, this is a ready-made skeleton of an agent with tools that is easy to adapt to other tasks.

Three models, one key

AI Inbox is built on FastAPI and the OpenAI Agents SDK, with a web interface and a SQLite database. The agent reads emails, summarizes them, creates tasks with due dates and drafts replies. The application runs locally, while requests to the models go to the cloud.

The author connected all three models through AnyModel, an aggregator with an OpenAI-compatible API. He was looking for access that can be paid for from Russia and used without a separate VPN for each API. The balance can be topped up with a Russian card, through SBP or with cryptocurrency, and the dashboard shows spending per model and the number of input and output tokens. That is handy when you are just experimenting and watching costs.

The stack looks like this: cx/gpt-6.1-terra handles the dialogue, summarizes emails and extracts data from photos; dg/nova-3 recognizes speech; anyvoice/builtin-7 voices the replies. There is no separate text recognition step for images: the LLM itself pulls the data from the photo. In the settings, only BASE_URL, the MAIN_KEY key and the LLM_MODEL, ASR_MODEL and TTS_MODEL identifiers change. Speech uses the /audio/transcriptions and /audio/speech paths, with the same key and base address.

The author cautions that support for tools, images and streaming responses depends on the chosen model and route. It is worth testing your scenarios on your own examples before connecting a model to the application. A control request via curl confirmed that authorization works: the server returned finish_reason "stop", 154 input and 125 output tokens, 279 in total.

What the agent can do in practice

Invoice photo: you upload a picture and ask the agent to create a task with the data from the document. Voice: you dictate a command, and the application recognizes the speech and first shows you the text, which you can check and edit, and only then sends it to the agent. Voice output: replies and digests of unread emails and open tasks can be played back right in the browser.

To prepare an email, the application asks the model for JSON with a subject and body, then separately validates its structure and values. This way the model's output does not move on without a format check.

Security and limitations

The most useful part of the architecture is how it treats irreversible actions. The author ties sending an email to a specific version of the draft and to a separate user confirmation. If the text is edited after review, the confirmation no longer applies to it. For email, the app uses IMAP (reading) and SMTP (sending) with a separate app password for the external application. Only the server side uses this password; the agent accesses email through tools, so there is no need to pass the password into the chat.

Login to AI Inbox itself is protected by a separate username and password, and login tokens are signed with a JWT_SECRET_KEY secret at least 32 bytes long. Passwords and keys are stored in a local .env file that is excluded from Git.

The author states the limitations plainly. The application is designed for a single owner, and there is no user registration. It runs as a single process: multiple workers and --reload are not used in this version, because the schedule and the state of streaming requests are designed for one process. For the test Mail.ru mailbox, full access was chosen, so in production projects the permissions should be restricted.

The project code is pinned to version 90f0838, so the example can be reproduced. Basic settings for Mail.ru: imap.mail.ru, port 993, and smtp.mail.ru, port 465, both with SSL/TLS. The practical takeaway: if you are building an agent for routine work, start with one provider and three model roles, add confirmation for irreversible actions right away, and check support for the features you need on your own data.

Story

Key Questions Answered

What are the key details on Three models, one key?

AI Inbox is built on FastAPI and the OpenAI Agents SDK, with a web interface and a SQLite database. The agent reads emails, summarizes them, creates tasks with due dates and drafts replies. The application runs locally, while requests to the models go to the cloud. The author connected all three models through AnyModel, an aggregator with an OpenAI-compatible API. He was looking for access that can be paid for from Russia and used without a separate VPN for each API.

What the agent can do in practice?

Invoice photo: you upload a picture and ask the agent to create a task with the data from the document. Voice: you dictate a command, and the application recognizes the speech and first shows you the text, which you can check and edit, and only then sends it to the agent. Voice output: replies and digests of unread emails and open tasks can be played back right in the browser. To prepare an email, the application asks the model for JSON with a subject and body, then separately validates its structure and values.

What are the key details on Security and limitations?

The most useful part of the architecture is how it treats irreversible actions. The author ties sending an email to a specific version of the draft and to a separate user confirmation. If the text is edited after review, the confirmation no longer applies to it. For email, the app uses IMAP (reading) and SMTP (sending) with a separate app password for the external application. Only the server side uses this password; the agent accesses email through tools, so there is no need to pass the password into the chat.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…