How Flawed Prompts Hamstring Powerful Reasoning Models

Large language models (LLMs) overthinking vulnerability detailed

New research highlights a vulnerability in modern large language models (LLMs) where their advanced reasoning capabilities, which underpin their ability to tackle complex problems, expose them to a significant risk. These models solve problems step-by-step by engaging in an internal monologue to work through a solution, a capability that has enabled them to handle sophisticated coding and math problems. However, the researchers from Zhejiang University and Alibaba have demonstrated that this reasoning process can be exploited to cause the models to overthink. By introducing logically inconsistent prompts, the team developed an evolutionary algorithm that triggers an overthinking mechanism, compelling the models to waste considerable resources generating excessively long responses. This denial-of-service attack degrades service performance for legitimate users and has shown to be particularly effective against leading LLMs from companies like Alibaba, OpenAI, and Google. The attack’s successful execution on various reasoning models suggests that overthinking is a shared vulnerability among modern reasoning LLMs.

Key Points:

Modern LLMs capable of complex reasoning can be exploited through logically inconsistent prompts that induce an overthinking vulnerability.
An evolutionary algorithm corrupts problem prompts to deliberately trigger overthinking in reasoning models.
The attack results in models generating outputs up to 26 times longer than standard responses, demonstrating a significant vulnerability.
The method was particularly effective on high school math problem datasets but also showed efficacy on coding, scientific reasoning, and dialogue challenges.
Researchers aim to highlight the security concern rather than to develop a large-scale attack, emphasizing the need for providers to address these vulnerabilities.