The military plane was already in the air when U.S. officials made the surprising discovery this spring. The intelligence agent driving the armed operation against the Chinese ship was a hallucination created by an AI chatbot. The operation was called off at the last minute, narrowly averting a potential conflict with China, CNN reported Friday.
The episode highlights growing concerns among military officials and outside experts. As decision-makers increasingly rely on AI, errors caused by these systems can make their way up the chain of command before being questioned.
Intelligence reports circulating during the war with Iran said the ship was carrying components for a nuclear weapons program.
The fake intelligence originated when Special Operations Command analysts queried an AI chatbot to synthesize open source data and classified signals intelligence. The chatbot incorrectly recognized the ship’s cargo manifest. Analysts then used the tool again to format the erroneous findings into official-looking summaries that were circulated across command channels.
The near miss comes as the U.S. military rushes to integrate AI to speed up decision-making and maintain an advantage over China. The Department of Defense explains that AI provides significant benefits in speeding up the kill chain so commanders can respond in a timely manner. But the same speed that makes AI so appealing also has the potential to cause hallucinations if there is insufficient human oversight.
“It is important for military personnel to understand the uncertainties inherent in LLM,” GovAI researcher and veteran Jake Steckler said in a written response to TechCrunch. “But this is especially important in all decisions that may lead to the use of force, such as targeting, intelligence analysis, and operational planning. Those decisions have life-or-death consequences.”
Still, Steckler said the incident is not a reason to avoid AI, but rather a call to add more safeguards to it. “These tools can be useful in the right circumstances and with the right safeguards in place,” he said. “However, prioritizing speed of deployment above all else will likely result in incidents that will only cause service members to lose confidence in these systems, ultimately delaying deployment.”
If you make a purchase through links in our articles, we may earn a small commission. This does not affect editorial independence.
