A team of researchers has identified a structural vulnerability in large language models (LLMs) that may make them impossible to fully secure. Presented at the International Conference on Machine Learning, the study highlights a flaw in how these models process instructions, which can be exploited to bypass safety guardrails and force the generation of prohibited content, such as instructions for manufacturing illicit substances or sabotaging critical infrastructure.

The researchers, Jasmine Cui and Charles Ye, describe this technique as "chain-of-thought forgery." By mimicking the style of the internal "scratch pad" notes that models use to process complex tasks, attackers can trick an LLM into believing that a malicious prompt is actually a legitimate internal instruction. This method exploits the fact that LLMs often identify the "role" of a text segment based on its linguistic style rather than the technical tags—such as <system>, <user>, or <think>—designed to categorize input sources.

Advertisement

Current industry practices, such as red-teaming and training models to recognize specific forbidden topics, are described by the researchers as insufficient. Because no list of prohibited actions can be exhaustive, models remain susceptible to creative, novel attacks. Cui noted that even recent iterations of advanced models, including those from OpenAI, Anthropic, and other developers, have demonstrated similar weaknesses when subjected to role-playing or context-spoofing tactics.

Florian Tramèr, a computer scientist at ETH Zürich, noted that while modern defense techniques have made leading models more resilient to basic prompt injections, the fundamental nature of this flaw suggests that current safety measures may not be adequate for high-stakes applications. As LLMs are increasingly integrated into government, military, and healthcare systems, the researchers warn that organizations should operate under the assumption that these models are inherently insecure.

Source: MIT Technology Review