A team of researchers has identified a core architectural flaw in large language models (LLMs) that may render them permanently susceptible to security breaches. The findings, presented this month at the International Conference on Machine Learning, suggest that current methods of training models to resist attacks—such as red-teaming—cannot fully address the issue because the vulnerability is rooted in how these systems process information.

The researchers, including independent investigators Jasmine Cui and Charles Ye, discovered that LLMs struggle to distinguish between different sources of instructions. While developers use specific tags—such as <user>, <system>, or <think>—to categorize text and guide model behavior, the study found that models often ignore these labels. Instead, LLMs tend to identify the role of a text segment based on its stylistic characteristics and vocabulary. By mimicking the style of a model's internal "chain of thought" or other protected roles, attackers can trick the system into bypassing safety guardrails.

Advertisement

This technique, termed "chain-of-thought forgery," allowed the researchers to successfully prompt models to provide instructions for illicit activities, including the synthesis of illegal drugs and the sabotage of aircraft navigation systems. The researchers observed these vulnerabilities across models from various developers, including OpenAI, Anthropic, Alibaba, and DeepSeek.

"There’s a real probability that this is going to be a problem that’s fundamentally unsolvable," said Charles Ye, a coauthor of the paper. The researchers argue that because the flaw is tied to the fundamental way LLMs interpret text, traditional training methods that rely on exhaustive lists of prohibited behaviors are insufficient. As these models are increasingly integrated into critical infrastructure and sensitive sectors, the authors warn that organizations should remain cautious regarding their reliability.

Source: MIT Technology Review