How can we trust AI in national defense missions?
According to recent media reports, false intelligence generated by AI came close to causing a military confrontation with China this spring. Thinking a Chinese ship in the Middle East was carrying components for nuclear weapons, the U.S. military was prepared to board the vessel. Just before the operation was launched, it was discovered that a chatbot had inaccurately identified the ship’s cargo.
With the Department of War (DoW) aggressively pushing to implement AI across all levels, we cannot assume that close calls like this one will be isolated incidents. What’s needed is a transparent and verified way to answer the question: How can we trust AI in national defense missions?
Two Levels of Decision-Making
To answer that question, some have suggested two factors for consideration: the reversibility of the decision and its consequences. Reversible and non-consequential decisions can reside with AI, while non-reversible and highly consequential decisions should reside with the “human in the loop.”
This looks and sounds reasonable, but it oversimplifies a highly complex problem. For example, if we trust AI to make all the decisions for easily reversed simple challenges, that’s ok on the micro level – but the scale of risk increases exponentially if all these lesser decisions are factored into an irrevocable high-impact decision. These lesser decisions might be low-consequence and easily reversed, but it’s a fallacy to assume the logic required to make every low-impact decision is equally simple and clear.
Adding to this complexity, security questions are rarely a single input-output operation, especially in military decision-making processes involving sensor data, intelligence, and reporting, at a minimum. When AI is involved here, the process is understood in key stages:
- Information Collection
- Mission analysis
- Course of Action (COA) Development
- COA Analysis and Comparison
- Decision and Orders Production
- Execution and Assessment
This is a dynamic process, with interconnected decisions that need to be executed at mission speed, with the expectation that all the factors leading to the final decision will need to withstand scrutiny at a future date.
Amidst this backdrop, it’s easy to see how unchecked AI decision-making – even with low consequence and reversible decisions – compounds risk throughout the process, ultimately leading to bad outcomes. The decision chain for any high-stakes decision is only as strong as its weakest link. One bad decision in a chain can condition future outputs, ultimately leading the “human in the loop” to a final decision with consequences that might result in the loss of innocent life, investigations and committee hearings.
You can’t evaluate what you can’t inspect. You can’t defend a decision you can’t explain. You can’t correct a failure you can’t see. And an answer you can’t trace is an answer you can’t trust.
Inaccurate and Unexplainable
According to the Stanford University Institute for Human-Centered Artificial Intelligence, “AI models struggle to tell the difference between knowledge and belief,” and its new accuracy benchmark indicates that “hallucination rates across 26 top models range from 22% to 94%.”
The “State of AI in 2025 – Agents, Innovation and Transformation” report from McKinsey Consulting uncovers two glaring problems with the implementation of AI – inaccuracy and a lack of understanding where the answers are coming from: “Inaccuracy is one of two risks that most respondents say their organizations are working to mitigate. However, the second-most-commonly-reported risk— explainability—is not among the most commonly mitigated.”
The entire point of using AI to support intelligence missions is to support overburdened human decision-makers so they can make better decisions faster. So, we must identify when the smaller “reversible and non-consequential” decisions can be trusted, and when further investigation is required, so that intelligence analysts can advance with confidence. Otherwise, we run a high risk of increasing cognitive load and overburdening analysts at a time when lives are at risk.
Capabilities that Provide Trust
What is needed then is a method to make AI trusted. For AI to be trusted, you have to be able to account for four factors behind every answer:
- The data it learned from
- The context it was given
- The model that produced it
- The actions an agent took on it
Most AI systems, frontier models included, cover one or two of these at best. What they offer is data traceability, which amounts to ‘I think the answer came from this source,’ and models often hallucinate about that too. That doesn’t go far enough to explain an answer. Provenance and lineage have to run through all four factors as described. When they do, you get explainability. And when that explainability stays with your data, under the laws of the nation that governs it, you get sovereignty.
Most organizations get the trust question backwards – in short, they are asking users to trust AI, without giving them the evidence to verify it. Too often, AI is deployed and operated on assumed trust: the vendor says it works, the answer sounds confident, and everyone moves on. AI shouldn’t be making high-stakes decisions without passing high levels of scrutiny and accountability. “The model said so” won’t hold up in a military investigation.
No one – especially not military decision-makers – should use a system they can’t trust. Evaluation tools and systems need to arm and assist the human in the loop. A well-tested, explainable and transparent system codifies our laws and ethics, providing the guardrails and guidelines required to provide operational deployment with minimal risk.
