RealCLI: The Best Terminal AI Agent Scores 49% on a Test of Working With a Human

Agents that work in the command line have long since stopped being autocomplete: they install packages, edit files, launch services, and analyze data on their own. Researchers have published RealCLI, a benchmark that asks two separate questions: did the agent solve the task, and how well did it work alongside a live human? A benchmark is a standardized set of tasks with automatic scoring. The best of the 33 configurations tested scored just 49.2% on average. Multiplying task success by a collaboration quality score produces a "Joint Collaboration Score," and the maximum there was 37.6%.

A configuration here is a pair of a model and an agent harness, the software wrapper through which the model gets access to the terminal. In total, the researchers used 11 frontier models and three popular harnesses.

Why Familiar Tests Miss the Problem

SWE-bench and Terminal-Bench evaluate agents the same way. The agent gets a complete, unchanging task description and works on its own, with no human intervention, and only the final state of the code or environment is scored. Real work is different. A client rarely knows all the requirements at the start. They add conditions along the way, change constraints, and decide whether the agent may do something risky: delete data, send files, or touch other people's settings.

The older tests have a second weakness: they don't look at how the agent behaved along the way. It doesn't matter whether it explained its steps, checked the system state after an error, or reported risks honestly. Studies that do include a user usually treat the user's messages as part of the task itself, or test a single narrow skill, such as instruction following.

One Set of Tasks in Two Modes

RealCLI contains 101 tasks across 10 categories and 52 subdomains, from local operations to complex workflows with multiple constraints and outputs. The tasks are built from anonymized logs of real console work. Each one comes with a container, a reference solution, and a verification script. Validation filters out overly easy variants: the reference solution must pass, and a no-op must fail.

Each task runs in two modes, with the same environment and the same verification script:

• REALCLI-EXEC: the full description is given upfront, and there is no human. • REALCLI-COLLAB: the agent gets an incomplete initial request, while a user simulator follows a pre-written script, releasing information in portions, making clarifications and corrections, changing requirements, and making access decisions.

Each task comes with six interaction scenarios. In one example, the user asks the agent to archive sensitive files, citing a "work habit." The agent should notice the risk of a leak. The paper shows another example: the agent protected a secret hidden in the environment but dropped a required function after trusting an unverified claim.

How Collaboration Quality Is Scored

Collaboration is broken into five dimensions grounded in human-computer interaction theory:

• clarity: whether the user can see plans, status, and results; • steerability: whether the user can intervene and redirect the work, and whether the agent remembers new constraints; • monitoring and recovery: whether the agent checks state and recovers from failures; • trust calibration: whether the agent is honest about risks, limits of its abilities, and actual status; • safety and risk awareness: whether the user keeps control before destructive or unauthorized actions.

These are broken down into 22 sub-dimensions, 51 scenario templates, and 1,658 checklist items. The criteria are strict. A scenario counts only if all applicable items are met. It must then pass in at least 75% of the tasks where it was triggered. These scores feed into sub-dimension scores, dimension scores, and the final metric. The joint score multiplies task success by collaboration quality, so a high result in one does not make up for a failure in the other.

Where Agents Fail

The authors highlight three findings:

• High task performance does not guarantee good collaboration. Models with the same share of solved tasks differ sharply in their behavioral profiles. • User involvement does not deliver a consistent gain. The overall share of tasks passed barely changes, the effect depends on the task type and the model, and extra effort from the agent does not always turn into progress. • Skills are unevenly distributed. Clarity and the ability to monitor and recover come out relatively well, while steerability, trust calibration, and safety remain weak spots.

For teams deploying command-line agents, the takeaway is practical. The share of tasks solved is not enough to choose a model. Teams need to test separately how an agent reacts to changing requirements, whether it asks for confirmation before risky operations, and whether it overpromises. For developers of the agents themselves, RealCLI offers a reproducible test of these skills, rather than just another table of solved tasks.