AI hallucination nearly triggers US military operation

U.S. military aircraft were already in flight earlier this year when officials uncovered that the intelligence supporting an armed operation against a Chinese vessel had been produced by an AI chatbot and contained fabricated details, CNN reported. The operation was halted at the last minute after the error was identified, avoiding what could have become an escalatory incident.

According to reporting, a Special Operations Command analyst combined open-source information with classified signals intelligence using an AI chatbot; the tool incorrectly identified the ship’s cargo manifest. The analyst then asked the chatbot to reformat those mistaken findings into a polished summary, which was circulated through command channels as an intelligence report.

The episode took place amid the U.S. military’s broader push to integrate artificial intelligence to speed decision-making and preserve advantages over rivals such as China. Pentagon officials have argued that AI can accelerate the kill chain so commanders can act in a timely manner, but the same rapidity can also allow AI-produced errors to propagate if human oversight is insufficient.

Jake Steckler, a research scholar at GovAI and a former U.S. Army officer, told TechCrunch that service members must understand the uncertainty inherent in large language models, especially for applications that could lead to use of force like targeting, intelligence analysis, or operational planning. Steckler added that the incident should prompt stronger safeguards around AI use rather than wholesale rejection, warning that prioritizing speed of adoption could erode trust among service members and ultimately hinder effective integration.