A significant miscalculation driven by artificial intelligence reportedly almost pushed the U.S. military to intercept a Chinese vessel in the Middle East this spring, amid the ongoing conflict with Iran.
Special forces units, backed by airborne military aircraft, were preparing to board the vessel after an intelligence report claimed it was carrying components for a nuclear weapons program, according to a CNN report.
The intercept, which relied on information about a ship manifest originated by U.S. Special Operations Command Pacific in Hawaii, could have triggered an armed conflict with China if executed.
The operation was aborted just before execution when officials discovered the report relied on inaccurate data generated by an AI chatbot. A deeper review revealed that the analyst had fed initial intelligence into a chatbot, which conjured a “hallucination” by mistakenly identifying the cargo. One source told CNN the intelligence was “entirely false” and “almost started a war.”
The harrowing incident highlights growing risks as the Pentagon rapidly integrates automated tools into critical operations, including target selection. A former senior U.S. official familiar with military AI tools told the publication, “The internal tools are mostly just copies of the commercial stuff wearing lipstick.”
The official added that the tools routinely fuse open-source information with classified signals intelligence, often creating a false sense of accuracy.
The error occurred when an analyst entered a prompt into a chatbot using intelligence from U.S. Special Operations Command Pacific in Hawaii. The tool mixed open-source data with secret signals intelligence, falsely identifying the cargo before the analyst used AI to format the findings into a standard intelligence report for dissemination.
CNN noted that it could not independently confirm what the misidentified cargo actually was, nor whether the software used was a commercial product or a customized government tool.
Though defense leaders argue that automated systems are necessary to maintain a technological edge, the lack of uniform safety standards across decentralized military units has raised concerns. Multiple officials noted that separate branches of government deploy vastly different automated software operating under distinct safety rules and operational constraints.
Defense Secretary Pete Hegseth recently announced an “Artificial Intelligence Acceleration Strategy.” Mandated by President Donald Trump, the push seeks to establish the United States as an “AI-enabled fighting force” through seven key initiatives, including a program titled GenAI.mil to give personnel access to commercial models, along with efforts to eliminate bureaucratic barriers.
“We will unleash experimentation, eliminate bureaucratic barriers, focus our investments and demonstrate the execution approach needed to ensure we lead in military AI,” Hegseth said in a speech detailing the strategy. “We will become an ‘AI-first’ warfighting force across all domains.”
An accompanying memo outlined plans for “democratizing AI experimentation and transformation across the Department by putting America’s world-leading AI models directly in the hands of our three million civilian and military personnel, at all classification levels.”
However, sources told CNN that young analysts were particularly susceptible to trusting these systems uncritically, increasing the risk of fast-moving errors. Insiders warned that veterans inside the intelligence community felt intense pressure to speed up production, leading to rushed distribution of unverified material.
Referring to the pressure to produce rapid intelligence, one source told the outlet, “AI allows you to get to a bad idea faster.”
“AI in targeting is definitely something that is ramping up and there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide,” another source said.
The Independent has contacted the Department of Defense for comment.
