AI guardrails hinder offensive cybersecurity research, researchers say

A new report reveals that AI guardrails implemented by companies like OpenAI and Anthropic are hindering the work of offensive cybersecurity researchers. These researchers, who search for unknown vulnerabilities and develop tools to exploit them, say that safety restrictions on AI models limit their ability to test and improve security systems.

The affected researchers are those who specialize in offensive cybersecurity, also known as ethical hacking or penetration testing. Their work involves finding flaws in software and networks before malicious actors can exploit them. However, when they use AI models to assist in this work, they often encounter guardrails that block or restrict their queries, slowing down their research.

This matters because offensive cybersecurity research is critical for identifying and patching vulnerabilities. If researchers cannot effectively use AI tools, it could leave systems more exposed to real attacks. The guardrails, designed to prevent misuse, are inadvertently impeding legitimate security work.

According to the report, researchers reported that AI models refused to generate code for exploits or provide information on known vulnerabilities, even when the intent was defensive. Specific examples include models blocking requests for proof-of-concept code or techniques used in penetration testing. The report did not provide exact numbers but highlighted multiple instances where guardrails interfered with research.

The background to this issue is the ongoing tension between AI safety and utility. Companies like OpenAI and Anthropic have implemented guardrails to prevent their models from being used for harmful purposes, such as generating malware or assisting in cyberattacks. However, these same restrictions can catch legitimate security research.

Looking ahead, the report suggests that AI companies may need to develop more nuanced guardrails that can distinguish between malicious and defensive uses. Some researchers are calling for special access or APIs for vetted security professionals. Without changes, the effectiveness of offensive cybersecurity research could be compromised, potentially increasing the risk of undiscovered vulnerabilities being exploited by malicious actors.