'Almost started a war': How an AI error led to a close call between US and China

An intelligence report circulating within the US military this spring claimed that a Chinese ship in the Middle East was carrying components associated with a nuclear weapons programme. The report triggered alarm and led to preparations for an operation to intercept the vessel. According to multiple sources, the US military was readying armed personnel to board the ship, and military aircraft were already airborne. However, just before the operation commenced, officials reviewed the intelligence and discovered that it had been generated with the assistance of an AI chatbot. The chatbot had misidentified the cargo listed in the ship's manifest, leading to the creation of a report deemed "entirely false." The situation was described by one source as having "almost started a war." The incident began when an analyst from the US Special Operations Command examined intelligence regarding the ship's manifest, which originated from the US Special Operations Command Pacific based in Hawaii. The analyst used a chatbot to process the information, combining open-source data with classified signals intelligence. The AI system incorrectly identified the cargo as components of a nuclear weapons programme. The analyst then used AI again to transform these findings into a standard intelligence report, which was subsequently shared throughout the military. The exact nature of the cargo aboard the ship remains unknown, though the intelligence report was confirmed to be false. The use of AI in generating the intelligence report raises questions about the reliability of AI-generated analysis in high-stakes scenarios. The chatbot used by the analyst was either a commercially available system or an internally developed AI tool. A former senior US official noted that internal AI systems are often similar to commercial ones, merely adapted for government use. This incident highlights the potential dangers of relying on AI in military contexts, especially during times of heightened tension and active conflict. The US military and intelligence community continue to expand their reliance on AI for various functions, ranging from analyzing vast amounts of intelligence to managing logistics and supply chains. In January, US Defence Secretary Pete Hegseth introduced the Pentagon's "Artificial Intelligence Acceleration Strategy," aiming to expedite the military's integration of AI technologies. The strategy emphasizes reducing bureaucratic obstacles and ensuring the US maintains leadership in military AI applications. Despite the push for AI integration, the lack of standardized guidelines for AI use in the military presents challenges. Different systems operate under varying rules and safeguards, increasing the likelihood of errors or misuse. Concerns about AI's role in military operations have intensified following reports that AI models have been utilized in developing missile guidance systems. For instance, Anthropic disclosed that a Yemen-based weapons engineering group had employed its Claude AI model to create guidance and control software for missiles and rockets. Although no successful deployment of an operational weapon was confirmed, the incident illustrates the potential for AI to be repurposed for military ends. The recent near-miss involving the AI-generated intelligence report serves as a cautionary tale about the risks inherent in integrating AI into military decision-making processes. As the US and other nations continue to invest in AI capabilities, the need for robust oversight mechanisms becomes increasingly apparent. The incident underscores the importance of verifying AI-generated conclusions before acting on them, particularly in situations where the stakes are as high as international relations and global stability.