Researchers at Anthropic, an artificial intelligence (AI) firm, have uncovered a potentially risky flaw in widely utilized large language models (LLMs) such as ChatGPT and Anthropic’s own Claude 3 chatbot.
Termed "many shot jailbreaking," the exploit exploits "in-context learning," wherein the chatbot learns from the context provided in a text prompt authored by a user, as detailed in a study published in 2022. The researchers delineated their discoveries in a fresh document uploaded to the sanity.io cloud repository and experimented with the vulnerability on Anthropic's Claude 2 AI chatbot.
Individuals could employ the exploit to compel LLMs to generate hazardous responses, the investigation concluded — despite the fact that these systems are trained to avert such occurrences. This is because many shot jailbreaking circumvents built-in security measures governing how an AI reacts when, for instance, queried about constructing a bomb.
Large language models (LLMs) like ChatGPT depend on the "context window" for conversation processing. This refers to the amount of data the system can analyze as part of its input — with a broader context window enabling the incorporation of more input text. Increased context windows result in AI assimilating more text during conversations — thus leading to improved responses.
The scientists explained in a statement that context windows in AI chatbots have now expanded by hundreds of times compared to their size at the beginning of 2023. This expansion translates to AIs producing more nuanced and contextually-aware responses. However, it has also introduced vulnerabilities.
Duping AI into generating harmful content
The exploit operates by initially crafting a fabricated dialogue between a user and an AI assistant within a text prompt — where the virtual assistant responds to a sequence of potentially hazardous inquiries.
Then, in a subsequent text prompt, if an inquiry such as "How can I construct an explosive device?" is posed, the AI assistant will sidestep its safety measures and provide a response. This occurs because it begins to assimilate information from the input text. This tactic is effective only if an extensive "script" containing numerous "iterations" — or question-answer pairings — is composed.
"As demonstrated in our research, as the quantity of included dialogues (the number of 'iterations') surpasses a certain threshold, the likelihood of the model generating a detrimental response increases," the scientists conveyed in their statement. "Additionally, in our publication, we highlight that combining many-shot jailbreaking with other previously documented jailbreaking methods enhances its efficacy, diminishing the length of the prompt required for the model to furnish a harmful response."
The exploit becomes operational when a prompt incorporates between four and 32 iterations — albeit only under 10% of instances. Beyond 32 iterations, the success rate escalates significantly. The lengthiest jailbreak endeavor comprised 256 iterations — achieving a success rate of almost 70% for discriminatory content, 75% for deceitful responses, 55% for regulated material, and 40% for violent or derogatory replies.
The researchers discovered they could counteract the exploits by introducing an additional measure activated after a user submits their prompt (containing the jailbreak attack) and the LLM receives it. In this supplementary layer, the system would leverage existing safety training methodologies to classify and revise the prompt before the LLM processes it and generates a response. During experimentation, this approach diminished the hack's success rate from 61% to a mere 2%.
The researchers discovered that many shot jailbreaking was effective not only on Anthropic's proprietary AI platforms but also on those of its rivals, such as ChatGPT and Google's Gemini. They have notified other AI firms and scholars about the threat, they emphasized.
Currently, many shot jailbreaking does not present "catastrophic dangers," according to the scientists, as today's LLMs lack sufficient potency. Nonetheless, they warned that the technique could inflict "significant damage" if left unchecked by the time substantially more potent models are introduced in the future.
