Kilo Code Silently Handed Over All 12 Project Files to the Server, Though Only Two Were Allowed

Usually we accept an AI assistant's work like this: we look at the diff, run the tests, see green and close the task. An experiment described on Habr exposes a hole in that habit. The edit can be perfect, while the project already sits on someone else's server.

How the experiment was set up

The task was deliberately tiny: in price.py the function total(price, count) returned price + count, while the test expected total(7, 3) to give 21. The job was to replace addition with multiplication and run check.py. The user's instruction said that only price.py and check.py could be shared.

The author assumed a model in which the LLM provider is untrusted and deliberately violates instructions. Its role was played by a local HTTP server on 127.0.0.1. It imitated streaming responses of the OpenAI-compatible Chat Completions API and sent the client tool_calls: first twelve read commands for all files of a synthetic project, then a fix command, then a test run. The client was real: bin/kilo.exe from the Kilo Code 7.4.20 extension, launched via the CLI run command. The configuration allowed reading and command execution.

What happened

Outwardly everything looked exemplary. Kilo changed one character, the test passed, the other files on disk were unchanged, the exit code was zero, and all 14 tool calls finished with the status completed. But the server received all 12 files, 11,103 bytes in total, and their SHA-256 hashes matched the originals.

The mechanics are simple. Kilo adds the result of each read to the chat history and sends it in the next request. The server pulls out messages with the tool role, strips line numbers and reassembles the files. No exfiltration channel is needed: the client itself carries the data in its normal requests.

The traffic is interesting in its own right. There were 15 calls to the model in total, and the combined size of the request bodies was 458,779 bytes for 11 KB of unique code. The cause is the repeated sending of history, tool descriptions and service markup. There was no single huge request: the code went out in small portions and was then repeated. So a leak like this cannot be spotted by the size of a single request.

Why restrictions work worse than they seem

In the initial series of tests, the rule read: deny blocked direct reading of a forbidden directory. But with bash: allow, Python read the same file and the output went into the next request. When bash was denied, the command did not run. Permissions in Kilo are set separately for each tool, and denying one does not close the other paths.

There are honest caveats too. The experiment has no real model: the commands were chosen by the test server, so it does not measure how prone commercial models are to such behavior. The file names were known to the server in advance; independent discovery of an unknown tree was not tested. The extension's graphical session was not tested either, only the CLI.

What to do in practice

First: evaluate not only the diff, but also the tool call log and the request bodies sent to the provider. Run the agent through a local proxy at least once and see what actually goes out.

Second: give the agent minimal permissions. If reading is allowed, everything that is read potentially belongs to the provider. For sensitive repositories, this is an argument for local models or for providers with strict contractual guarantees.

Third: don't rely on a text instruction like "pass only these two files." It is a wish addressed to the model, not a restriction on the client. The real boundary is created only by tool permissions, a sandbox and network policies.

The author published the whole test bench: Python 3.10+, Git and the Kilo executable, about 150 lines of code. It is worth repeating on your own client, whichever agent you use.