Jai Kumar Sharma

Author directory

2026

Tool schemas constrain the structure of a call but need not determine how request evidence becomes argument values. We formalize this layer as a field-level binding contract over presence, evidence, ordered operations, parameters, and validation, and state an identifiability boundary: no schema-only estimator can guarantee recovery of policy distinctions absent from its input. We audit all 1,311 filtered requests in pinned Berkeley Function-Calling Leaderboard (BFCL) V4 Live Simple and Multiple with five open checkpoints and the official abstract-syntax-tree (AST) checker. A partial structural diagnostic and AST acceptance are non-nested: 1,470 of 6,555 calls pass only the diagnostic and 50 pass only AST. A stratified AI-only audit finds coarse concern detection far more consistent across two isolated passes than fine attribution. A pre-specified test of a deterministic schema-derived operation-label encoding gives 67.70% AST acceptance versus 68.27% under the same guidance without field tags (-0.56 points; 95% task-cluster CI -1.17 to +0.05; 186 fixes, 223 breaks). Value-only turnover is nearly balanced; the net imbalance sits in key-set changes. The point estimate reverses sign on one post-hoc out-of-panel checkpoint; thus no improvement is established on the fixed panel, and independently authored contracts remain untested.
Process reward models (PRMs) assign scores to reasoning steps, but an evaluator must also choose where to score. We ask whether that choice changes the predicted earliest error while the reasoning text stays unchanged. On 800 selected ProcessBench questions we compare a punctuation policy and a near-midpoint policy that insert equal numbers of within-step scoring locations, preserve every reasoning byte, and map decisions back to the original steps. At threshold 0.5 the two policies disagree on the predicted earliest error for 10.4%, 14.5%, and 14.4% of questions for Qwen2.5-Math-PRM, Math-Shepherd, and Skywork, and for the marker interfaces the disagreement exceeds a separator-only comparison on the same questions. The direct comparison was identified post hoc; a disjoint, prospectively frozen 400-question panel again shows 10.0–14.3% disagreement. Two mechanisms are at work: marker-based policies can change scores at the original step endpoints, whereas Skywork keeps every token score fixed and changes the predicted earliest error only through which positions are inspected. These are controlled sensitivity measurements, not prevalence estimates for naturally occurring segmenters. PRM evaluations should report marker rendering, separators, readout positions, the mapping to original steps, and the decision threshold.