Pentagon sets procedures for AI-assisted software development

A new instruction approved by Pentagon CIO Kirsten Davies includes procedures for AI-assisted software development as the department adapts for an era of “software-defined warfare.”

The guidance on “Accelerated Mission Software” was signed by Davies on Aug. 31 and went into effect Sept. 8.

The far-reaching issuance sets policy, responsibilities and procedures for software modernization and management within the department and encourages enterprisewide software reuse.

“The DoW recognizes that AI-assisted software development is a significant force multiplier for the delivery of speed and quality,” officials wrote in the 37-page document, using an acronym to refer to the Department of War, which is the Trump administration’s preferred name for the Department of Defense.

The instruction states that DOD components should employ AI technologies to modernize software systems, boost operational efficiency and accelerate capability delivery, noting that the tech can be used to automate software development processes, optimize system integration, and improve resilience and security.

The new directive comes amid concerns in the tech community and elsewhere about the pace of AI development and the risks to humanity associated with those advancements.

The DOD instruction was issued to “establish the security, accountability, and transparency baseline that enables teams to confidently adopt AI in mission-critical software development contexts while managing associated risks,” officials wrote. The document suggested that personnel could be on the hook if things go badly.

“Developers and development teams remain fully accountable for the security, functionality, and integrity of any code generated or modified using AI,” officials wrote.

AI-generated changes to security- or safety-critical functionality must be reviewed and approved by a human, they noted.

“AI-generated code will be considered unverified input, and its use does not absolve the developer or the government of responsibility for the resulting work product,” per the instruction.

The document noted that all code “suggested or generated by AI” will also be subject to the same types of review and security testing processes as manually written code.

“Code must be explicitly reviewed for security vulnerabilities, safety implications, logical errors, subtle bugs, potential intellectual property infringement, license obligations, and proper implementation of security controls before being integrated into a codebase,” according to the guidance.

DOD components are tasked with determining use case-specific risk and mitigation requirements for each AI-enabled capability, verifying that performance benchmarks and risk remediation efforts are sufficient prior to deployment, and ensuring that performance benchmarks are continuously monitored while the capabilities are in operation.

“To ensure transparency when using AI, software efforts will maintain a record of models, versions, and significant datasets used to generate or test software. This information will be included as part of the comprehensive software evidence package, analogous to the SBOM, to enable risk assessment and traceability of AI-driven components,” officials wrote.

The guidance also calls for the use of digital capabilities that can detect and flag “unintended bias” in AI systems that personnel need to address.

The new instruction reflects Pentagon officials’ concerns about the potential cyber leakage of sensitive information amid a boom in commercially available generative AI tools. Things like non-public DOD code, configuration scripts, infrastructure definitions, schematics, or documentation may not be entered into or processed by generative AI applications or services that don’t reside on the department’s information systems or haven’t been approved for use by Pentagon personnel, per the edict.

With regard to approved applications or services for the department’s proprietary software development, officials want contractual guarantees that government data and user prompts aren’t shared or used for training any public or “DoW-external models.”

DOD components are directed to ensure that “appropriate members of development teams” get trained on the principles for AI-enabled software development that were outlined in the instruction and that they have “awareness of common AI-generated vulnerability patterns, best practices for prompt engineering to elicit secure code, and understanding the risks of intellectual property and license contamination.”