Some difficulties in learning Chinese as a second language (L2 Chinese) involve choosing a sentence pattern that is grammatically well formed but less natural in context. Mandarin ba/SVO choice is one such case: learners may avoid ba when the discourse favors it or extend it to contexts where SVO is more appropriate. Because edit-based correction systems may leave both alternatives unchanged, diagnosing these choices requires sensitivity to discourse context, making the task a targeted test of context-sensitive natural language understanding (NLU). Using 200 controlled discourse contexts, we examine whether autoregressive LLM surprisal can serve as a graded diagnostic of this distinction. Across 21 open-weight models, sentence-level mean surprisal identifies the contextually preferred continuation with a mean model-level accuracy of 87.93%, whereas a MacBERT-based correction baseline leaves most contextually non-preferred continuations unchanged. Relevant context generally lowers surprisal for the intended continuation, and sentence-level scoring is more reliable than first-token scoring, whose interpretation varies with tokenizer granularity. The results suggest that surprisal can complement edit-based correction in L2 pragmatic assessment, while cross-model asymmetries, structural preference biases, and tokenizer effects underscore the need for construction-specific controls and human interpretation when evaluating context-sensitive NLU.

With the prosperity of large language models (LLMs), powerful LLM-based intelligent agents have been developed to provide customized services with a set of user-defined tools. State-of-the-art methods for constructing LLM agents adopt trained LLMs and further fine-tune them on data for the agent task. However, we show that such methods are vulnerable to our proposed backdoor attacks named BadAgent on various agent tasks, where a backdoor can be embedded by fine-tuning on the backdoor data. At test time, the attacker can manipulate the deployed LLM agents to execute harmful operations by showing the trigger in the agent input or environment. To our surprise, our proposed attack methods are extremely robust even after fine-tuning on trustworthy data. Though backdoor attacks have been studied extensively in natural language processing, to the best of our knowledge, we could be the first to study them on LLM agents that are more dangerous due to the permission to use external tools. Our work demonstrates the clear risk of constructing LLM agents based on untrusted LLMs or data. Our code is public at https://github.com/DPamK/BadAgent