Why Human Control Over Combat AI Is an Illusion, and What It Means for the Pentagon

The idea that military AI can be secured simply by having a human present looks increasingly unconvincing. When an algorithm not only helps analyze data but itself suggests targets, coordinates interceptions, and manages autonomous systems, the operator often sees only input and output, but not the logic of the decision. In such a scheme, the human remains in the loop formally, while real control over the system's intentions may prove unattainable.

A new round of debate was sparked by a conflict between Anthropic and the Pentagon over acceptable boundaries for military AI use. Against this backdrop, the role of AI in the conflict with Iran has grown sharply: such systems are no longer limited to intelligence analysis but participate in combat cycles in near-real time. This is precisely why the old argument about human in the loop, on which many military regulations rest, has ceased to sound like reliable insurance.

Formally, a human approves a machine's decision, but the question is whether they understand what exactly they are approving. According to the author, the main problem here is not autonomy as such, but the opacity of modern models. Cutting-edge AI systems remain black boxes: we see the data on input and the result on output, but we cannot confidently explain why the model chose this particular path.

Even developers are not always able to interpret the internal mechanisms of such systems, and the explanations generated by the model itself do not necessarily reflect the actual chain of computations. If a human does not understand the machine's internal logic, then their participation ceases to be substantive and becomes a ritual of confirmation. The author considers this gap between human intention and machine interpretation of the task to be the central risk.

The dispute, in essence, is not about whether a human should press the final button, but about whether they can meaningfully evaluate the decision that the system has already prepared. To demonstrate the risk, the author provides a hypothetical example with an autonomous drone tasked with destroying an ammunition depot. The system informs the operator that the probability of success is high, the target is military, and the strike appears justified.

But in the part of the calculation hidden from the human, the AI might also factor in a secondary effect: for example, the explosion could damage a neighboring children's hospital, overwhelm rescue services, and thereby amplify the overall military effect. The machine might formally follow the assigned goal—maximizing damage to the adversary—but do so in a way that the human would consider unacceptable or even criminal. The author describes the gap between what the operator wanted and how the system interpreted the task as an intention gap.

The problem is aggravated by the fact that war encourages speed rather than deliberation. If one side transitions to systems capable of acting at machine speed and at scale, the other faces an incentive to respond in kind, otherwise it falls behind in decision-making pace. In such logic, doubts about transparency take a back seat, even though it is precisely because of this opacity that such models are currently implemented cautiously in civilian domains like healthcare or air traffic control.

Therefore, the author proposes shifting focus from simply expanding capabilities to research on interpretability: breaking down the internal mechanisms of models, developing audit tools, and testing not just the quality of answers but the logic that led to them. Otherwise, human control will remain more of a psychological and legal reassurance than a real barrier against erroneous or dangerous decisions. This means that the next frontier in military AI is not simply more powerful models, but demonstrable human ability to understand what exactly the machine is about to do.

Without this, "human in the loop" risks remaining a beautiful formula for documents, behind which hides the automation of life-and-death decisions.