Experts worry about transparency, unforeseen risks as DOD forges ahead with new frontier AI projects

Pentagon leadership recently tapped four major tech companies for separate contracts — each worth up to $200 million — to accelerate the Defense Department’s enterprise-wide adoption of some of the most advanced commercial algorithms and machine learning capabilities and deploy them against contemporary national security challenges.

Also called foundation models, frontier AI refers to sophisticated, constantly evolving, cutting-edge systems that are becoming increasingly intelligent at completing tasks like natural language processing, computer vision and reasoning. 

They’re rapidly pushing the boundaries of what existing AI can achieve. 

But while experts have warned about the unknowns associated with frontier AI development creating risks to humanity, the Defense Department’s Chief Digital and AI Office (CDAO) is not responding with full transparency regarding if and how those four powerful models were vetted to ensure they are safe for responsible operational use, before the high-dollar contracts were awarded.

In an emailed response to multiple questions DefenseScoop asked the CDAO regarding any early efforts conducted to demonstrate the models could be trusted for DOD-specific applications, a defense official said:

“The contract awards to frontier AI companies are designed to enable the department to leverage the technology and talent at these companies to develop agentic AI workflows across a variety of mission areas. DOD will tailor technical approaches based on mission need and industry capabilities. As part of the protoyping effort, DOD will assess both the opportunities and risks of frontier AI models in DOD use cases. The DOD is committed to ensuring that all deployments of AI technologies comply with applicable security policies and executive orders. In our prototyping and experimentation efforts with frontier AI companies, we are exercising risk management practices throughout the technology lifecycle to safeguard our people, our data, and our mission, while preventing unauthorized use.”

The official declined to provide more information beyond that statement in response to follow-up and clarification questions.

Not enough info

During his 36 years of service in the U.S. Air Force, now retired Lt. Gen. Jack Shanahan accumulated more than 2,800 flight hours. He went on to work in DOD’s Intelligence and Security directorate, before the Joint Artificial Intelligence Center (JAIC) was formed in 2018 and he was tapped to serve as its first chief. The JAIC was one of the original DOD organizations that were fused to form the CDAO in 2022.

“I love talking about test and evaluation. When I was at the JAIC, I would always make it abundantly clear — we’re not going to talk about anything sensitive, I’m not going to give the media access to exactly what we’re doing — but I owe it to people to talk about this. Why would we not?” Shanahan told DefenseScoop in a recent interview. 

The defense official’s responses, in his view, should have included a more straightforward acknowledgement that the CDAO is or will work closely with each of the four companies — OpenAI, Anthropic, Google and xAI — to understand their T&E and red-teaming processes used to prove and refine the models.

“Where I think you gave them a softball and they swung and missed at it was, rather than just saying anything at all about test and evaluation, they said ‘risk management practices.’ That’s a missed opportunity. I understand why somebody would use that phrase, but why not just say ‘Risk management is part of T&E … so we’re going to partner closely with these companies, and we’re going to get this thing right?’” Shanahan said. “It is as simple as that.”

Further, he would’ve liked to see more information regarding whether the vendors supplied DOD with their raw model weights, which essentially encapsulate all that the AI systems have learned in training and ultimately represent the core intelligence of each model.

“If they shared those raw model weights with the government, that’s a big deal, because then the government can do a lot more than just getting access to the model itself,” Shanahan said.

In a separate discussion with DefenseScoop about the CDAO’s recent foundation model awards, AI safety engineer Dr. Heidy Khlaaf pointed out that T&E and related risk assessments typically take significantly longer time than the timescales observed for the four contracts. 

Khlaaf currently serves as the chief AI scientist at the AI Now Institute, where she concentrates on the assessment and safety of AI within autonomous weapon systems.

“The DOD recently cutting the size of the Office of the Director of Operational Test and Evaluation in half speaks for itself. In a lot of ways, there is signalling for much faster AI adoption without the rigorous processes that have existed since the 1980s to ensure new technologies are safe or effective,” she said.

Pointing to publicly available information regarding the four commercial models and latest evaluation results, Khlaaf argued that they would likely not meet the standard defense thresholds expected for systems to be used in critical military-supporting settings.  

“We’ve particularly warned before that commercial models pose a much more significant safety and security threat than military purpose-built models, and instead this announcement has disregarded these known risks and boasts about commercial use as an accelerator for AI, which is indicative of how these systems have clearly not been appropriately assessed,” Khlaaf explained.


There are certain contracts, such as experimental use cases and research and development projects, that might not require T&E or risk assessments. However, Khlaaf noted, such checks would be exceedingly necessary in the CDAO’s current frontier AI efforts — as the announcement explicitly calls out the use of “AI capabilities to address critical national security challenges.”

“An independent assessment to substantiate these companies’ claims has always been an existing core requirement of military and safety-critical systems, and it guarantees that no aspect of the system’s pipeline is compromised, while ensuring a system’s security and fitness for use,” she said.

Existing, relevant risks that accompany discarding T&E practices, Khlaaf added, were already evident in a recent viral incident where Elon Musk-owned xAI’s model — Grok — praised Adolf Hitler, referred to itself as MechaHitler, and generated other antisemitic content. 

“This was due to an updated system prompt by the Grok team itself to nudge it towards a specific view. It dispels the myth that frontier AI is somehow objective or in control of its learning. A model can always be nudged and tampered by AI companies and even adversaries to output a specific view, which gives them far too much control over our military systems. And this is just one security issue out of dozens that have been unveiled over the last several years that have yet to be addressed,” Khlaaf told DefenseScoop.

‘Not risk-free’

Shanahan pointed out that the new AI Action Plan issued by President Donald Trump days after the CDAO announced the frontier AI partnerships “says explicitly, ‘evaluate frontier AI systems for national security risk.’”

Drawing from his personal experiences, the former fighter pilot said it is important to consider the different sets of risks between prompting commercial frontier AI capabilities on one’s home computer, versus applications inside the Pentagon.

“They’re going to be used potentially for intelligence analysis, for the development of operational plans and courses of action. And maybe only a subset of [these use cases] will be true, life or death [warfare-type applications], but there will be serious consequences if and when these models confabulate, get things wrong or spit garbage out the other end,” he said.

The Pentagon’s awards to the four companies will be made under indefinite-delivery, indefinite-quantity (IDIQ) contracts that will be paid out as a now undetermined amount of the services and technology — worth up to $800 million across the four companies — are purchased and delivered based on adapting demands during a fixed period of time. 

The defense official declined to directly answer DefenseScoop’s questions about whether the CDAO has made any awards under the four IDIQ deals, to date. 

Meanwhile, Shanahan expressed concerns that Pentagon officials may already be accessing the foundation models inside the building, even as it is against current policies.

“I promise you, people are using these in their lives in the Pentagon, even though they haven’t officially been allowed to do that. But that’s different than saying, ‘OK, I’m going to develop an intelligence assessment and an intel analysis using one of these models, and I’m going to give that forward information.’ Well, what if the data [that trained the model] was corrupted? What if China had access to that data and the model was spitting out something exactly the opposite of what it should have been? Who knows that?” he said.

Taken one way, the response to DefenseScoop’s questions might suggest that officials plan to assess the risks of each model based on each experimental use case they run, he noted.  

“Play around with it. That’s great — I’m all for it. But if you’re going to use these operationally, you’ve got to have a level of confidence [that they are safe]. And how do you get that? Well, it’s a combination of the companies sharing with you their own internal testing and their benchmarks that they did, but also the government’s ability to do this themselves,” Shanahan said.

According to Khlaaf, not conducting T&E sets a precedent that paves the way for faster adoption without the due diligence that systems are ensured to have an accepted minimal baseline of safety and security guarantees. And on a technical level, without proper T&E, military data and operations can be threatened with experimental uses. 

“If, for example, critical military data is used to fine-tune these models, inherent vulnerabilities within LLMs can allow for the extraction of this critical data through observed model predictions alone by unapproved parties. Other attack vectors include poisoning web-scale training datasets and ‘sleeper agents’ within commercial foundation models that compromise their supply chain, which may only be triggered during specific instances to intentionally or inadvertently subvert models used within military applications and compromise their behavior,”
Khlaaf said. “So unfortunately experimental use is not risk-free, especially without preliminary T&E to ensure that such experimentation would in fact be risk-free.”

And once the models are deployed in the wild beyond prototyping, even those used for the most banal applications that are often associated with bureaucratic functions — like communications, coding, resolving IT tickets and data processing — can introduce threats. 

The speed DOD takes to deploy them holds potential for compromising the safety of civil and defense infrastructure, because administrative tasks can feed into mission-critical decisions. 

“But as repeated research [efforts] have shown, AI tools consistently fabricate outputs — known as hallucinations — and introduce novel vulnerabilities,” Khlaaf said. “Their use might lead to an accumulation of errors, and over time small errors will propagate to intelligence or decision-making, which could result in decisions that cause civilian harm and tactical mistakes.”

OpenAI, Google and xAI did not respond to DefenseScoop’s requests for more information about the new frontier AI partnerships with CDAO.

An official from Anthropic did not provide details about T&E conducted with or for DOD, but said that the company conducts “rigorous safety testing” — and that it was “one of the first AI labs to receive ISO/IEC 42001:2023 certification for responsible and safe AI,” which marks an early international standard for AI Management Systems. 

Anthropic’s “Claude models are the most resistant to prompt injection and jailbreak techniques (CalypsoAI model security leaderboards, Huggingface) and are the least likely to hallucinate models on the market (MASK leaderboard),” the official said in an email. They noted that Anthropic will continue to work with its commercial cloud partners to ensure that its models are available while meeting the DOD’s most stringent requirements for information-handling at the controlled unclassified level and above. 

“On model weights, we do not share our model weights,” the Anthropic official told DefenseScoop.