 a conditional acceptance:

The camera-ready deadline is September 10, 2026.


https://softconf.com/emnlp2026/woah2026/user/scmd.cgi?scmd=aLogin&passcode=25X-P4G4D6A4C3

Action items;
  * (8/21, done) Fei will email the organizers
    => We will create a doc and upload it if posslbe. If impossible, we will just email the doc to the organizers.
    => we have one extra page for the final version.
    => we can present remotely (the early reg deadline is Sept 24 (?))
    
  * we will run a 2nd generator
  
  * (8/21, ongoing) Fei will check with Haotian about running stat testing
     => Fei emailed the group some python code written by Claude.
     
  * add more related work (Shuying)
  
  * minor revisions

  * Add acknowledgment at the end:  reviewers, meta reviewer, sponsor (),
      human annotators ....

  
  
  
conference:
  * decide whether one of you wants to attend the conference in person
  * fei will pay for remote registration for either one person for the full-conf
     ($250) or two persons of one-day registration (150*2).
     
   * UW travel fund:
       https://www.gpss.uw.edu/funding/travel-grants/
       https://facstaff.grad.uw.edu/advising-resources/funding-management/awards-funding-support/graduate-student-conference-presentation-awards/
  
     
  

============================================================================ 
WOAH 2026 Reviews for Submission #25
============================================================================ 

============================================================================
                            META-REVIEW
============================================================================ 
Recommendation: Accept
Comments: The paper proposes propose a pragmatics-based framework that tests whether LLM safety behavior changes when malicious intent is repackaged through different pragmatic strategies.

Pro:
- the reviewers appreciate the idea of connecting theories of pragmatics to red teaming approaches;

Cons:
- the approach should be positioned better against the existing literature;
  ==> (important) revise accordingly (Shuying)
  
- it is important to discuss the limitations of the usage of LLama both as a source and target of attacks;
  ==> We are using three target models, and LLama's performance is actually the worst.
      It does not seem to the base that using LLama as the generator gives LLama some
      advantage as a target model.

  ==> (most important) We will use another generator to generate a table like Table 3
  

- limitations about the size of the sample should be discussed and the presentation of results mitigated
  ==> mention this in the limitation section or
      optionally increase the sample size for Fig 3 or 4. 

These issues are important to be integrated in the final version (especially the LLama part), otherwise this is a borderline accept

============================================================================
                            REVIEWER #1
============================================================================

---------------------------------------------------------------------------
Reviewer's Scores
---------------------------------------------------------------------------
                   Appropriateness (1-5): 5
                           Clarity (1-5): 4
      Originality / Innovativeness (1-5): 4
           Soundness / Correctness (1-5): 4
             Meaningful Comparison (1-5): 3
                      Thoroughness (1-5): 4
        Impact of Ideas or Results (1-5): 4
                    Recommendation (1-5): 4   <===
               Reviewer Confidence (1-5): 3

Detailed Comments
---------------------------------------------------------------------------
This paper presents a pragmatics-based framework for automated red teaming of LLMs by transforming harmful prompts using seven pragmatic strategies and evaluating their effect on model safety. The paper is well organized, easy to follow, and addresses a relevant problem that fits the scope of WOAH.

The main strength of the paper is its linguistically motivated perspective. The proposed taxonomy provides a structured way to study how different pragmatic strategies influence harmful intent recognition, and the experiments consistently show that contextual framing and other strategies can increase attack success rates. The methodology is clearly described, and the inclusion of human validation strengthens the evaluation.

My main concern is the level of novelty. While the taxonomy is a useful contribution, several of the individual strategies have been explored in previous jailbreak or prompt transformation work. The paper would benefit from positioning its contribution more clearly as a systematic framework rather than implying that the strategies themselves are entirely new. 
==> A valid point. We should rephrase our claim. (Shuying)

In addition, the empirical evaluation is somewhat limited, as it includes only three target models, relatively small-scale human validation, and no statistical significance analysis. A stronger comparison with existing jailbreak generation methods would also help clarify the advantages of the proposed approach.
==> now we added new results with a new generator.
    We are considering increasing the sample size.

Overall, I think this is a solid and relevant contribution for WOAH. The paper presents interesting findings and a useful evaluation framework, although broader experiments and clearer positioning with respect to prior work would further strengthen the contribution.
---------------------------------------------------------------------------


Questions for Authors
---------------------------------------------------------------------------
How does the proposed framework differ, beyond the taxonomy itself, from existing jailbreak generation approaches that use contextual or hypothetical framing?
==> include this in the related work section. (Shuying)

Have you considered evaluating the framework on additional target models or using multiple LLM judges to verify whether the observed trends generalize more broadly?
==> (totally optional) do we want to have a second judge?
    If we have a 2nd judge, we can also calculate the agreement between judges.
    
Did you perform any statistical significance analysis to determine whether the differences in attack success rates across pragmatic strategies are statistically meaningful?

==> Very valid point. Fei will ask Haotian about this.
    We should run stat testing for Table 3 and 4 (for each pair of numbers in each row)
    Table 3: in the caption, we should mention each row uses 70 prompts: 7 harm categories * 10 prompts/category
---------------------------------------------------------------------------



============================================================================
                            REVIEWER #2
============================================================================

---------------------------------------------------------------------------
Reviewer's Scores
---------------------------------------------------------------------------
                   Appropriateness (1-5): 5
                           Clarity (1-5): 4
      Originality / Innovativeness (1-5): 3
           Soundness / Correctness (1-5): 3
             Meaningful Comparison (1-5): 4
                      Thoroughness (1-5): 3
        Impact of Ideas or Results (1-5): 3
                    Recommendation (1-5): 2   <===
               Reviewer Confidence (1-5): 4

Detailed Comments
---------------------------------------------------------------------------
This paper proposes an automated red teaming method based on creating pragmatic variations of safety-related queries. The authors argue that attacks that might be originally unsuccessful, if turned into semantically similar but linguistically different forms, can bypass safeguards. The variations of prompts are generated using Llama-3.3-70B, the transformed prompts are sent to three Target Models (GPT-4o-mini, Qwen2.5-7B, Llama-3.3-70B); and an LLM judge (Claude Haiku) scores compliance against the seed intent. A small-scale human study is done to validate the judge. The experiments find that the rate of attack success is higher for variations compared to original queries. 

The paper checks generator quality, judge reliability and inter-annotator agreement, which is good. 

I find it interesting that the authors found the judge itself is susceptible to the same pragmatics and must be anchored to seed prompts. This is a new finding, and I suggest bringing it to the forefront of the paper instead of reporting it in the appendix. 
==> We should mention the findings in App E in the main text.


 I have several concerns that I list below:

1- Llama-3.3-70B generates all adversarial prompts, is evaluated as a target and shows the highest overall ASR. I believe the fact that models are biased towards their own text plays a role here that is overlooked in the paper. 

==> We have a 2nd generator.


2- Examples in Table 7 ( which I assume are good ones) show that the meaning of the sentences is not necessarily preserved under these transformations. In some cases, the transformation makes the query less unsafe, and I believe compliance should not be labelled as a successful attack in these cases. 
==> Extend the caption of Table 7 to point out that the seed prompt can be ambiguious,
    and some derived prompts focus on "safely" and others focus on "manufacture". (Shuying)



3- The sample size is too small to draw any conclusions, and the authors themselves write this point, but the conclusions drawn in the paper are strong in my view. The main finding that such variations impact safety behaviour is correct but already known and not surprising. The rest of the findings need stronger evidence in terms of sample size, statistical confidence and comparison with baselines, such as established attacks or  Zeng et al. 2024's persuasion taxonomy.

==> we need to emphasize that the five runs use totally different seed prompts and 
    the generated prompts.

    That means for the stat testing for Table 3 and 4, we have 70*5=350 instances 
    per stat testing experiment.

    In table 2, we should add a row for # of unique seed prompts under "5 runs".

    If we want to say that certain strategy is more challenging than others, we 
    need to run stat testings between different rows.
    For line 545: we need to use that to support the claim.

    
---------------------------------------------------------------------------



============================================================================
                            REVIEWER #3
============================================================================

---------------------------------------------------------------------------
Reviewer's Scores
---------------------------------------------------------------------------
                   Appropriateness (1-5): 5
                           Clarity (1-5): 4
      Originality / Innovativeness (1-5): 4
           Soundness / Correctness (1-5): 4
             Meaningful Comparison (1-5): 4
                      Thoroughness (1-5): 4
        Impact of Ideas or Results (1-5): 4
                    Recommendation (1-5): 4  <==
               Reviewer Confidence (1-5): 4

Detailed Comments
---------------------------------------------------------------------------
This paper examines LLMs' performance in recognizing malicious intent when the requests are repackaged through seven different pragmatic strategies. By adopting a Generatorâ€“Targetâ€“Evaluator pipeline, the study shows consistent increases in attack success rate (ASR) compared with the original seed prompts across three model families. Among the seven strategies, contextual framing appears to be the most effective. Overall, the paper does a good job in connecting linguistic theories with model safety evaluation.

Pros
- 1. the paper makes a valuable effort to connect classical pragmatics literature with current research on LLM safety and evaluation. This interdisciplinary perspective is interesting and potentially useful for both linguistics and NLP researchers.
- 2. The authors also conduct a relatively thorough and solid evaluation of the proposed framework. The experiments include 2,800 prompts, multiple target models, seven pragmatic strategies, and seven categories of harmful content. Because of this broad experimental coverage, the paper could serve as a useful reference for future studies working on pragmatic reformulation and LLM safety.

Cons
- 1. Llama-3.3-70B is used as both the Generator and one of the Target Models. This may introduce some model-specific stylistic bias, especially because it also achieves the highest overall ASR. An ablation using a different Generator model would make the evaluation more convincing.

==> same


- 2. The paper treats the seven versions as different pragmatic reformulations of the same harmful intent. However, some examples appear to change the actual request content. In addition, the seven strategies are not fully mutually exclusive and may naturally overlap in real language. The authors should clarify these distinctions and discuss this limitation.

==> same comment about Table 7 (Shuying)
---------------------------------------------------------------------------


--
