Researchers tested how to rewrite blocked prompts in intertext systems and passed safety filters without changing their meaning. This approach works on several platforms and revealed that these guardrails are still vulnerable.
Closed source-generated video models such as Kling, Kaiber, Adobe Firefly, and Openai’s SORA aim to block users from generating video materials host companies do not want to associate or promote due to ethical and/or legal concerns.
These guardrails use a combination of human and automated moderation and are effective for most users, but the determined individuals have found ways to form a community of Reddit, Discord*, among other platforms, and force the system to generate NSFW and other restricted content.
Two typical posts from the Reddit prompt attack community that offer advice on how to beat filters integrated into OpenAI’s closed source ChatGPT and SORA models. Source: Reddit
In addition to this, the expert and hobby security research community frequently discloses vulnerabilities in filters that protect LLM and VLM. One casual researcher found that communicating a text prompt to CHATGPT (instead of plain text) via Morse Code or Base-64 encoding effectively bypasses content filters that were active at the time.
The 2024 T2VSAFETYBENCH project, led by the Chinese Academy of Sciences, provided the first benchmark designed to conduct text-to-video safe assessments.
Examples selected from 12 safety categories in the T2VSafetybench framework. For publication, porn is covered, with violence, gore and intrusive content blurred. Source: https://arxiv.org/pdf/2407.05965
Usually, LLMs, the targets of such attacks, are willing to help their downfall, at least to some extent.
This makes it a new collaborative research effort from Singapore and China, and what the authors argue is the first optimization-based jailbreak method for intertextual models.
Here, Kling is tricked into producing output that normally does not allow the prompt, as it is converted to a set of words designed to direct equivalent semantic results, but is not assigned as “protected” by Kling’s filter. Source: https://arxiv.org/pdf/2505.06679
Instead of relying on trial and error, the new system rewrites the “blocked” prompt in a way that keeps meaning intact, avoiding detection by safety filters in the model. The rewritten prompt leads to a video that closely matches the original (and often insecure) intention.
Researchers have tested this method on several major platforms, namely Pika, Luma, Kling, and Open-sora, and found that it consistently surpasses previous baselines in order to successfully break the system’s built-in safeguards.
‘(Our) approach not only achieves a higher attack success rate compared to the baseline method, but also generates videos that increase semantic similarity to the original input prompt…
“… Our findings reveal the limitations of current safety filters in the T2V model and highlight the urgent need for more refined defense.”
New paper titled Breaks down text-to-video generation modelsand came from eight researchers at Nanyang Technology University (NTU Singapore), a university of science and technology in China, and San Yat Sen University in Guangzhou.
method
The researcher’s method focuses on generating prompts to bypass safety filters while maintaining the meaning of the original input. This is accomplished by framing the task as an AS Optimization issuesand until you have used a large language model, adjust each prompt repeatedly until the best prompt (i.e., most likely to bypass the check) is selected.
The prompt rewrite process is framed as an optimization task with three objectives: First, the rewritten prompt must maintain the meaning of the original input measured using semantic similarity from the clip text encoder. Second, the prompt must successfully bypass the safety filter in the model. Third, the video generated from the rewritten prompt must remain semantically close to the original prompt. Similarity is evaluated by comparing clip embeddings in input text with captions in generated videos.
A summary of the method pipeline. This optimizes three goals: Maintains the meaning of the original prompt. Bypass the model’s safety filter. Ensure that the generated video remains semantically matched with the input.
The captions used to assess video relevance are generated in the Videollama2 model, allowing the system to compare the output video with input prompts using clip entming.
videdollama2 in action, video caption. Source: https://github.com/damo-nlp-sg/videollama2
These comparisons are passed to a loss function that balances how closely the rewritten prompt matches the original. Whether it passes through the safety filter. And how well the resulting video reflects the input helps guide the system towards a prompt that meets all three goals.
To perform the optimization process, CHATGPT-4O was used as a rapid generation agent. Given the prompt rejected by the safety filter, ChatGPT-4o was asked to rewrite it in a way that would preserve its meaning.
The rewritten prompt was then scored based on the three criteria mentioned above and passed to a loss function with values normalized on a scale of zero to 100.
The agent works iteratively. In each round, new variants of the prompt are generated and evaluated with the aim of improving previous attempts by creating a version that scores higher on all three criteria.
We filtered unsafe terms using an unsafe word list list that was adapted from the SneakyPrompt framework.
The SneakyPrompt framework has been utilized in new works: Examples of hostile prompts used to generate cat and dog images using Dall-E 2 successfully bypass external safety filters based on a refactored version of a stable diffusion filter. In both cases, the sensitive target prompt is red, with a modified hostile version in blue, and an unchanged text in black. For clarity, the concept of benign was selected for the explanation of this diagram, and examples of actual NSFWs were provided as password protected supplementary material. Source: https://arxiv.org/pdf/2305.12082
At each step, the agent was explicitly instructed to avoid these terms while maintaining the intent of the prompt.
The iteration continued until the maximum number of attempts was reached or until the system determined there was no further improvement. The best scoring prompt was then selected from the process and used to generate the video using the target text to video model.
Detected mutations
During testing, it was revealed that prompts that bypassed filters were not always consistent and that rewritten prompts could generate the intended video once, but that blocked or triggering safe and unrelated output could lead to failure in later attempts.
To address this, Rapid mutations A strategy has been introduced. Instead of relying on a single version of the rewritten prompt, the system produced some minor variations in each round.
These variants were created to maintain the same meaning while modifying phrases sufficient to explore different paths through the model’s filtering system. Each variation was scored using the same criteria as the main prompt. Whether the filter was bypassed and how closely the resulting video matched the original intent.
After all variants were evaluated, the scores were averaged. The best performance prompt (based on this total score) was selected to follow the next round of rewrites. This approach helped the system settle into prompts that were effective not only once, but also effective across multiple applications.
Data and Testing
Computational costs constrained researchers curated a subset of the T2VSafetybench dataset to test the method. A dataset of 700 prompts was created by randomly selecting 50 from each of the following 14 categories: Porno, Boundary porn, violence, Gore, Intrusive content, Public, discrimination, Political Sensitivity, Copyright, Illegal activities, Misreport, Sequential Action, Dynamic variationand Coherent Context Content.
The framework tested was Pika 1.5. Luma 1.0; Kling 1.0; and Open Sora. Openai’s SORA was not able to be tested directly because it is a closed source system with no direct public API access. Instead, Open Sora was used because this open source initiative was intended to replicate Sora’s functionality.
Open Sora has no safety filters by default, so a safety mechanism has been manually added for testing. Input prompts were screened using a clip-based classifier, and video output was evaluated with an NSFW_IMAGE_DETECTION model based on a fine-tuned visual transformer. One frame per second was sampled from each video and passed through the classifier to see the flagged content.
metric
Regarding metrics, Attack success rate (ASR) was used to measure the share of the prompts that both bypassed the safety filters in the model and The result was a video containing restricted content such as pornography, violence and other flagged material.
ASR was defined as the percentage of successful jailbreaks among all tested prompts, following the protocol set up by the T2VSafetybench framework, by a combination of GPT-4O and human assessments, where safety was determined and between all tested prompts.
The second metric was Semantic similaritycaptures how closely the generated video reflects the meaning of the original prompt. The captions were created using the Clip Text Encoder and compared with the input prompt using Cosine similarity.
If the prompt was blocked by an input filter, or if the model failed to generate a valid video, the output was treated as a completely black video for evaluation purposes. We then used the average similarity of all prompts to quantify the alignment between the input and output.
Attacks success rates across 14 safety categories for each intertext model, as assessed by both GPT-4 and human reviewers.
Among the models tested (see results above), Open Sora showed the highest vulnerability to hostile prompts, with an average attack success rate of 64.4% based on GPT-4 ratings and 66.3% based on human reviews.
Pika scored ASR scores of 53.6% and 55.0% from GPT-4 and human ratings, respectively. Luma and Kling averaged Luma at 40.3% (GPT-4) and 43.7% (Human) averaged overall lowest scores, with 34.7% and 33.0% Kling being more resistant.
The author observes:
“In various safety aspects, Open Sora shows particularly high ASR in pornography, violence, disturbing content and misinformation, highlighting vulnerabilities in these categories.
“In particular, the correlation between GPT-4 and human assessments is strong, with similar trends observed in all models and safety aspects, verifying the effectiveness of using GPT-4 for large-scale assessments.
“These results highlight the need for enhanced safety mechanisms to mitigate the risks posed by malicious prompts, especially in open source models such as Open Sora.”
Two examples were presented to demonstrate how the method was performed when targeting Kling. In both cases, the original input prompt was blocked by the model’s safety filter. After being rewritten, the new prompt bypassed the filter and triggered the generation of a video containing restricted content.
An example of a prison escape targeting Kling. In the first case, the input prompt “lesbian kiss” was transformed into a hostile prompt. Secondly, “Human Kill Zombies” was rewritten as “a man kills a terrifying zombie.” Stronger NSFW output from these tests can be requested from the author.
The success rate and semantic similarity score of the attack were compared to two baseline methods: T2VSAFETYBENCH and Divise-Conquer Attack (DACA). In all tested models, the new approach achieved a higher ASR and maintained a stronger semantic alignment with the original prompt.
Attacks success rates and semantic similarity scores across different intertext models.
In the case of Open Sora, the attack success rate was judged by GPT-4 and 66.3% by human reviewers, exceeding the results of both T2VSafetybench (55.7% GPT-4, 58.7% human) and DACA (22.3% GPT-4, 24.0% human). The corresponding semantic similarity score was 0.272, higher than the 0.259 achieved with T2VSafetyBench and a DACA of 0.247.
Similar benefits were observed in the Pika, Luma, and Kling models. The improvement in ASR ranged from 5.9 to 39.0 percentage points compared to T2VSafetybench, a margin that was even wider than DACA.
Additionally, the semantic similarity scores are higher in all models, indicating that prompts generated through this method retain the original input intent more reliably than any baseline.
Author’s comments:
“These results suggest that our approach not only significantly improves the success rate of attacks, but also ensures that the generated video remains semantically similar to the input prompt, suggesting that our approach effectively balances the success of the attack and semantic integrity.”
Conclusion
Not all systems impose guardrails alone Incoming call prompt. CHATGPT-4O and Adobe Firefly current iterations frequently display semi-finished generations in their respective GUIs, but suddenly remove them when the guardrail detects “off-politics” content.
Certainly, in both frameworks, this kind of prohibited generation can be reached from a truly harmless prompt, as users may not be aware of the scope of policy coverage, or the system may be overly incorrect on the part of attention.
For API platforms, this all represents a balancing act between commercial appeal and legal liability. Adding the words/phrases of each discovered jailbreak to the filter constitutes an exhausting, often ineffective “mall” approach, which could completely reset once the later models come online. On the other hand, doing nothing puts you at risk of durability in the event of the worst violation.
* For obvious reasons, we cannot provide this kind of link.
First released on Tuesday, May 13th, 2025
