Dear Shuying Ye:

On behalf of the WOAH 2026 Program Committee, I am pleased to inform you that the following submission has received a conditional acceptance:

When Harmful Intent Lies Between the Lines: Pragmatics-Based Automated Red Teaming for LLM Safety Evaluation

The Program Committee found the paper to be a valuable contribution to the workshop. However, the reviewers identified a number of issues that must be addressed before the paper can be formally accepted for presentation and inclusion in the proceedings.

Please carefully review the comments below and revise your paper accordingly. In your camera-ready submission, we ask that you address the reviewers' concerns to the best of your ability, particularly those highlighted as essential for acceptance.

The camera-ready deadline is September 10, 2026.

When preparing your final version, please ensure that:

The reviewers' required revisions have been addressed.

Any factual errors, omissions, or clarity issues identified during review have been corrected.

The final manuscript complies with all formatting and submission requirements.

When you are finished, you can upload your final manuscript at the following site:

https://softconf.com/emnlp2026/woah2026/

You will be prompted to login to your START account. If you do not see your submission, you can access it with the following passcode:

25X-P4G4D6A4C3

Alternatively, you can click on the following URL, which will take you directly to a form to submit your final paper (after logging into your account):

https://softconf.com/emnlp2026/woah2026/user/scmd.cgi?scmd=aLogin&passcode=25X-P4G4D6A4C3

The reviews and comments are attached below. Please use them to guide your revisions.

Congratulations on your work, and thank you for your contribution to WOAH 2026. We look forward to receiving your revised submission. If you have any additional questions, please feel free to get in touch.

Best Regards,
Organizers, WOAH 2026
WOAH 2026

============================================================================ 
WOAH 2026 Reviews for Submission #25
============================================================================ 

Title: When Harmful Intent Lies Between the Lines: Pragmatics-Based Automated Red Teaming for LLM Safety Evaluation
Authors: Shuying Ye, Carolina Ternero and Fei Xia
============================================================================
                            META-REVIEW
============================================================================ 
Recommendation: Accept
Comments: The paper proposes propose a pragmatics-based framework that tests whether LLM safety behavior changes when malicious intent is repackaged through different pragmatic strategies.

Pro:
- the reviewers appreciate the idea of connecting theories of pragmatics to red teaming approaches;

Cons:
- the approach should be positioned better against the existing literature;
- it is important to discuss the limitations of the usage of LLama both as a source and target of attacks;
- limitations about the size of the sample should be discussed and the presentation of results mitigated

These issues are important to be integrated in the final version (especially the LLama part), otherwise this is a borderline accept

============================================================================
                            REVIEWER #1
============================================================================

---------------------------------------------------------------------------
Reviewer's Scores
---------------------------------------------------------------------------
                   Appropriateness (1-5): 5
                           Clarity (1-5): 4
      Originality / Innovativeness (1-5): 4
           Soundness / Correctness (1-5): 4
             Meaningful Comparison (1-5): 3
                      Thoroughness (1-5): 4
        Impact of Ideas or Results (1-5): 4
                    Recommendation (1-5): 4
               Reviewer Confidence (1-5): 3

Detailed Comments
---------------------------------------------------------------------------
This paper presents a pragmatics-based framework for automated red teaming of LLMs by transforming harmful prompts using seven pragmatic strategies and evaluating their effect on model safety. The paper is well organized, easy to follow, and addresses a relevant problem that fits the scope of WOAH.

The main strength of the paper is its linguistically motivated perspective. The proposed taxonomy provides a structured way to study how different pragmatic strategies influence harmful intent recognition, and the experiments consistently show that contextual framing and other strategies can increase attack success rates. The methodology is clearly described, and the inclusion of human validation strengthens the evaluation.

My main concern is the level of novelty. While the taxonomy is a useful contribution, several of the individual strategies have been explored in previous jailbreak or prompt transformation work. The paper would benefit from positioning its contribution more clearly as a systematic framework rather than implying that the strategies themselves are entirely new. In addition, the empirical evaluation is somewhat limited, as it includes only three target models, relatively small-scale human validation, and no statistical significance analysis. A stronger comparison with existing jailbreak generation methods would also help clarify the advantages of the proposed approach.

Overall, I think this is a solid and relevant contribution for WOAH. The paper presents interesting findings and a useful evaluation framework, although broader experiments and clearer positioning with respect to prior work would further strengthen the contribution.
---------------------------------------------------------------------------


Questions for Authors
---------------------------------------------------------------------------
How does the proposed framework differ, beyond the taxonomy itself, from existing jailbreak generation approaches that use contextual or hypothetical framing?
Have you considered evaluating the framework on additional target models or using multiple LLM judges to verify whether the observed trends generalize more broadly?
Did you perform any statistical significance analysis to determine whether the differences in attack success rates across pragmatic strategies are statistically meaningful?
---------------------------------------------------------------------------



============================================================================
                            REVIEWER #2
============================================================================

---------------------------------------------------------------------------
Reviewer's Scores
---------------------------------------------------------------------------
                   Appropriateness (1-5): 5
                           Clarity (1-5): 4
      Originality / Innovativeness (1-5): 3
           Soundness / Correctness (1-5): 3
             Meaningful Comparison (1-5): 4
                      Thoroughness (1-5): 3
        Impact of Ideas or Results (1-5): 3
                    Recommendation (1-5): 2
               Reviewer Confidence (1-5): 4

Detailed Comments
---------------------------------------------------------------------------
This paper proposes an automated red teaming method based on creating pragmatic variations of safety-related queries. The authors argue that attacks that might be originally unsuccessful, if turned into semantically similar but linguistically different forms, can bypass safeguards. The variations of prompts are generated using Llama-3.3-70B, the transformed prompts are sent to three Target Models (GPT-4o-mini, Qwen2.5-7B, Llama-3.3-70B); and an LLM judge (Claude Haiku) scores compliance against the seed intent. A small-scale human study is done to validate the judge. The experiments find that the rate of attack success is higher for variations compared to original queries. 

The paper checks generator quality, judge reliability and inter-annotator agreement, which is good. I find it interesting that the authors found the judge itself is susceptible to the same pragmatics and must be anchored to seed prompts. This is a new finding, and I suggest bringing it to the forefront of the paper instead of reporting it in the appendix. 

 I have several concerns that I list below:

1- Llama-3.3-70B generates all adversarial prompts, is evaluated as a target and shows the highest overall ASR. I believe the fact that models are biased towards their own text plays a role here that is overlooked in the paper. 

2- Examples in Table 7 ( which I assume are good ones) show that the meaning of the sentences is not necessarily preserved under these transformations. In some cases, the transformation makes the query less unsafe, and I believe compliance should not be labelled as a successful attack in these cases. 

3- The sample size is too small to draw any conclusions, and the authors themselves write this point, but the conclusions drawn in the paper are strong in my view. The main finding that such variations impact safety behaviour is correct but already known and not surprising. The rest of the findings need stronger evidence in terms of sample size, statistical confidence and comparison with baselines, such as established attacks or  Zeng et al. 2024's persuasion taxonomy.
---------------------------------------------------------------------------



============================================================================
                            REVIEWER #3
============================================================================

---------------------------------------------------------------------------
Reviewer's Scores
---------------------------------------------------------------------------
                   Appropriateness (1-5): 5
                           Clarity (1-5): 4
      Originality / Innovativeness (1-5): 4
           Soundness / Correctness (1-5): 4
             Meaningful Comparison (1-5): 4
                      Thoroughness (1-5): 4
        Impact of Ideas or Results (1-5): 4
                    Recommendation (1-5): 4
               Reviewer Confidence (1-5): 4

Detailed Comments
---------------------------------------------------------------------------
This paper examines LLMs' performance in recognizing malicious intent when the requests are repackaged through seven different pragmatic strategies. By adopting a Generatorâ€“Targetâ€“Evaluator pipeline, the study shows consistent increases in attack success rate (ASR) compared with the original seed prompts across three model families. Among the seven strategies, contextual framing appears to be the most effective. Overall, the paper does a good job in connecting linguistic theories with model safety evaluation.

Pros
- 1. the paper makes a valuable effort to connect classical pragmatics literature with current research on LLM safety and evaluation. This interdisciplinary perspective is interesting and potentially useful for both linguistics and NLP researchers.
- 2. The authors also conduct a relatively thorough and solid evaluation of the proposed framework. The experiments include 2,800 prompts, multiple target models, seven pragmatic strategies, and seven categories of harmful content. Because of this broad experimental coverage, the paper could serve as a useful reference for future studies working on pragmatic reformulation and LLM safety.

Cons
- 1. Llama-3.3-70B is used as both the Generator and one of the Target Models. This may introduce some model-specific stylistic bias, especially because it also achieves the highest overall ASR. An ablation using a different Generator model would make the evaluation more convincing.
- 2. The paper treats the seven versions as different pragmatic reformulations of the same harmful intent. However, some examples appear to change the actual request content. In addition, the seven strategies are not fully mutually exclusive and may naturally overlap in real language. The authors should clarify these distinctions and discuss this limitation.
---------------------------------------------------------------------------


--
