Army moves to assess AI’s ‘unpredictable behaviors’ and safeguard autonomous systems
The Army tapped a research and development company to create software that can identify and analyze “unpredictable” artificial intelligence behavior to ensure its AI-enabled systems are trustworthy, according to an award notice posted Thursday.
Army Contracting Command awarded an other transaction agreement worth approximately $6.3 million to Advanced Technology International, Inc. as the “managing prime” to Battelle Memorial Institute, for technology that can evaluate potential risks associated with AI-enabled autonomous systems and their models.
While a relatively small award, the notice references a little-known program meant to help the Army identify risky AI behavior as the Pentagon pushes various models to the force. AI experts inside and outside the government have noted that the technology can behave unpredictably and sometimes dangerously, and they’ve advocated for safeguards.
The new deal is for the Generative Unwanted Activity Recognition and Defense (GUARD) prototype project, which intends to detect unpredictable AI behavior, “ultimately ensuring that these next generation autonomous capabilities are trustworthy and effective for future military operations,” officials wrote in the notice.
Army Contracting Command and Advanced Technology International, Inc. did not respond to DefenseScoop’s request for comment by publication.
DefenseScoop reported in 2024 that the Army was working on a “responsible AI strategy” based on Pentagon guidelines. The effort sought to define its own AI ethics and safety policy as the service pursued new high-tech capabilities.
The Defense Department Chief Digital and Artificial Intelligence Office’s webpage on “AI assurance” — a phrase that encompasses AI risk management, safety and trustworthiness — remains blank, simply saying “coming soon.”
However, a recently updated DOD manual from the Director of Operational Test and Evaluation noted “tracking of system safety and unexpected behavior” for AI-enabled and autonomous systems was a “major shift” it was assessing.
The ATI contract, however small, could signal a boost in the military’s interest in defining guardrails for its systems by identifying and analyzing unpredictable AI behavior, at least for the Army.
According to the notice, the GUARD program uses “advancements in neural network explainability, AI cognitive research, and game AI development to create Behavior-Event Graph data structures. These translate policies and behaviors into a network of inputs, actions, and events allowing analysis of potential emergent behavior before, during, and after field tests.”
Other prototype activity is expected to include things such as custom launchers and mobile containers for ground-launched capabilities, warhead solutions and customization, component fabrication, and other parts “to tune lethality, impulse, and fragmentation signatures while improving manufacturability.”
The OTA awardee is expected to develop GUARD software that can be used to create AI “risk profiles,” according to the notice.
The Army anticipates that if ATI can successfully develop this software, it will award further contracts without competition from other vendors, per the notice. It also specified that the $6.3 million figure is the anticipated total value of the prototype “if all milestones are exercised.”
