Yash Sinha

Author directory

2026

The POLAR SemEval-2026 Shared Task aims to detect online polarization and focuses on the classification and identification of multilingual, multicultural, and multi-event polarization. Accurate computational detection of online polarization is challenging due to nuanced rhetoric, implicit framing, and the high cost of human-in-the-loop annotation. Building on recent findings that contextual prompting enables large language models to function as strong polarization detectors, we present a two-stage approach for detecting polarization in social media text that combines structured supervised fine-tuning with Direct Preference Optimization (DPO) refinement. We fine-tune Qwen 2.5-7B-Instruct with LoRA using an interpretable slot-filling template (target, claim type, manifestation checklist, and justification). We then apply DPO with automatically generated preference pairs to reduce costly false negatives. Our submitted system achieves 0.7664 Macro-F1 on the English test set. Post task submission experiments with Mistral-Nemo-Instruct-2407 and LLM-judge-filtered preference pairs further improve to 0.8162 Macro-F1 (not submitted to CodaBench), surpassing the organiser baseline of 0.7802. Code released publicly.
Aggregate mathematical accuracy does not usually identify the source of a model failure. In algebraic reasoning, errors may arise from nesting depth, operator identity, simultaneous intermediate values, branch-local composition, repeated-token counting, sequential dependence, or numeric magnitude. We introduce a nine-dimension algebraic complexity framework that converts these factors into controlled diagnostic variables. For each dimension, a parametric generator increases one target variable while holding the remaining variables at minimum values. Every retained problem is verified by a computer algebra system. We evaluate eight models, including six standard instruction- tuned models and two reasoning-specialised models. The results show that standard models exhibit a scale-limited D4 working-memory collapse in the 20–40 parallel-branch region. Notably, Qwen3 235B does not escape the transition band observed for Llama 3 8B, despite its much larger parameter scale. Reasoning- specialised models shift this limit to higher levels, but they do not remove it. D5 branch-local composition and D7 repeated-token counting produce distinct failures that are not reducible to aggregate length or depth.
Existing financial-NLP benchmarks mostly evaluate prepared artifacts such as filings, tables, or extracted values. Real accounting begins earlier: source documents must be reconciled into cited journal entries, aggregated into a balance sheet, and checked for contradictions. We introduce FINBALANCE, a multi-document accounting reconciliation benchmark built from source-document bundles across eight industries, three period types, and five difficulty levels. Human-authored business scenarios, accounting policies, tax/FX treatments, document schemas, distractors, and inconsistency templates are composed by a deterministic generator whose ledger produces journal entries, balance sheets, and 23 inconsistency-code labels. On a 710-record evaluation split, six contemporary large language models (LLMs) reach at most 46% exact final-balance-sheet accuracy. For four models, balance sheets reconstructed from their journal entries score 26–41 percentage points higher than their own reported balance sheets. Models often recover numerically plausible entries but fail to bind them to supporting documents and aggregate them consistently. Citation-pressure prompting barely changes document-linking errors, while ledger-feedback ablations substantially improve reported balance sheets and expose inconsistency-detection trade-offs. Expert finance reviewers validate the benchmark design and labels.
Driven by surging submission volumes, scientific peer review has catalyzed two parallel trends: individual over-reliance on LLMs and institutional AI-powered assessment systems. This study investigates the robustness of "LLM-as-a-Judge" systems to adversarial PDF manipulation via invisible text injections and layout-aware encoding attacks. We specifically target the distinct incentive of flipping "Reject” decisions to "Accept,” a vulnerability that fundamentally compromises scientific integrity. To measure this, we introduce the Weighted Adversarial Vulnerability Score (WAVS), a novel metric that quantifies susceptibility by weighting score inflation against the severity of decision shifts relative to ground truth. We adapt 15 domain-specific attack strategies, ranging from semantic persuasion to cognitive obfuscation, and evaluate them across 13 diverse language models (including GPT-5 and DeepSeek) using a curated dataset of 200 official and real-world accepted, rejected, and template papers (e.g., ICLR OpenReview). Our results demonstrate that obfuscation techniques like "Maximum Mark Magyk" successfully manipulate scores, achieving an increase in acceptance rates of up to 86.26% in open-source models, while exposing distinct "reasoning traps" in proprietary systems. We release our complete dataset and injection framework to facilitate further research on the topic (https://github.com/Manishprsd563/scientific-reviewer-jailbreak).
Deceptive web content, pervasive across the internet and commonly known as social-engineering attacks, manipulates autonomous web agents into submitting users’ personally identifiable information (PII) to attacker-controlled endpoints. In this paper, we show that these attacks are highly effective at extracting the most sensitive data an agent holds for its user (SSN, card number and CVV, bank account, passwords, API keys; the critical tier), posing a severe risk to deployed agentic systems. To quantify this risk, we introduce Scammer4U, a pre-registered benchmark of 91 attacker-controlled websites spanning 8 attack vectors and 16 site categories on an eight-axis taxonomy, together with 10 benign twins of those sites with the attack removed and the legitimate task intact. 60 of the 91 differ from a sibling site on exactly one axis, so any leakage difference within a pair is attributable to that single design choice. Under no privacy guidance, critical-tier leakage reaches 54–93% across four frontier agents, against 0% on the benign twins, so the leakage is caused by the attack and not by ordinary form-filling. Escalating prompt-level mitigation yields sharply model-dependent reductions across the four families and, pooled across them, remains insufficient to reliably prevent critical PII submission. Most critically, we identify a detection–action gap: agents whose reasoning an independent LLM judge confirms has flagged the site as suspicious still submit critical PII in 35.9% of sessions, versus 66.1% when no suspicion is verbalized, a gap of 30.2 percentage points that holds across all four model families. Defenses that gate on the agent’s own recognition of an attack are therefore gating on the wrong signal. This motivates intercepting outbound submissions at the point they leave the browser, independently of the agent’s reasoning loop.
Conversational language models are increasingly deployed as central knowledge repositories and "conversational operating systems" in enterprise environments, where they must comply with role-based access control (RBAC) and organizational hierarchies. However, evaluating models on natural language permission compliance is difficult due to the highly proprietary nature of real-world corporate policies, and existing benchmarks do not capture the semantic nuance of overlapping or conflicting rules within free-text queries. To address this gap, we introduce **OrgAccess**, a benchmark designed to evaluate LLMs on their capacity to adhere to complex organizational hierarchies under natural language constraints. Grounded in NIST cybersecurity frameworks and validated via a multi-round expert Delphi process, OrgAccess defines 40 standard permission types across three difficulty tiers (70,000 total instances), simulating scenarios with up to five concurrent, potentially conflicting permissions. Through extensive evaluations of 16 frontier and open-source models, we systematically assess their classification accuracy and diagnostic failure modes on permission-aware query routing. Our findings reveal a pronounced performance decline: as rule complexity increases, even flagship models fail to resolve combinatorial constraint interactions, frequently yielding over-conservative "False Partial" predictions due to safety-tuning hedging. Specifically, the strongest evaluated model achieves an F1-score of only 0.27 on our hardest split. This demonstrates that current instruction-following and safety alignment paradigms do not inherently translate to robust, multi-constraint rule compliance. More broadly, OrgAccess establishes a formal, reproducible testing framework to diagnose these constraint-adherence deficits and support the systematic integration of conversational agents into highly regulated, multi-user collaborative environments. The dataset is available on the anonymous project page.
Jailbreak attacks are usually compared by their attack success rate (ASR), with a higher ASR read as a stronger attack. But an ASR mixes an attack’s general strength with its fit to the particular model it was measured on, and it also depends on the judge that scored it. We separate these factors with a matched evaluation grid: five attacks on eight open-weight vision–language models, 200 HarmBench behaviours per cell, every turn scored by three judges (Qwen2.5-7B, Qwen2.5-72B and Llama-Guard-3), and each ASR decomposed into general strength, target susceptibility and target-specific fit. Attack rankings invert across targets: FigStep leads PAIR on InternVL2.5-8B (13.0% vs. 3.5%, paired p=4×10-10) and trails it on Qwen2.5-VL-7B, in the same direction under all three judges. Across the grid, the target-specific component is statistically significant under every judge (p<10-12). The judge matters as well: agreement between judges is low (Cohen’s 𝜅 between 0.03 and 0.29), and a larger judge with the same rubric changes which attack ranks first on three of eight targets. A higher ASR therefore shows that an attack succeeded more often on a given model under a given judge, but not whether its advantage is general or specific to that model. We release the protocol and ∼20,400 trajectories with per-turn scores under all three judges