Maksim Silchenko

Author directory

2026

Reference-free quality estimation (QE) metrics see only the source and the hypothesis, so a translation that silently drops source content leaves nothing for them to flag. QEbreak, a pre-registered contrastive challenge set submitted to the WMT26 Automated Translation Quality Evaluation task (Subtask 4), measures this blind spot: 2,886 segments over 10 language directions, built as matched {base, omission, addition} triples, a fact-free verbosity ladder, and numeric-contradiction pairs as a positive control. Every hypothesis, endpoint, and equivalence margin was frozen in git-timestamped commits before any score existed. Scores from 31 systems confirm the deficit and refute our mechanism. Reference-free QE metrics under-penalize omission (family asymmetry A = −0.51; a current QE baseline scores the omission at or above its own base in 19.5 percent of matched contests, strictly above in 14.3), and the edit-direction by reference-availability interaction is significant (β = −0.069, 95% CI [−0.081, −0.057]). But reference-based neural metrics are asymmetric in the same direction (A = −0.42), outside the pre-registered ±0.30 equivalence margin, so omission under-penalization belongs to learned metrics as a class; reference access softens it but does not remove it. Every returned metric penalizes pure verbosity, and two systems score corrupted numerals above correct ones.
This paper describes the thylinao system for the ChildSafeAds shared task on monitoring sponsored segments in child-facing YouTube videos. The system is a six-member ensemble: two fine-tuned encoders, a zero-shot prompted Qwen3-32B, a LoRA-tuned Qwen3-14B used for ST2 only, a TF-IDF baseline, and a level-2 stacker for the compliance sub-task, blended at equal weight behind a decision layer whose every parameter is fitted on channel-grouped out-of-fold predictions. The system scores mean macro-F1 0.6537 on the evaluation split; inference over the 503 test instances is dominated by one measured 14-minute GPU pass of the 32B member. The zero-shot member is the most valuable and nearly the cheapest, while removing the fine-tuned DeBERTa changes nothing measurable. An ablation over the task’s four data access levels, run on the zero-shot member, finds the channel name adds nothing and the product-page crawl helps product identification but not, measurably, compliance. Validating these changes also surfaced properties of the evaluation, reported as feedback: the metric’s divisor dominates score variance and is recoverable in closed form from a published baseline row, and the benchmark is channel-disjoint but substantially advertiser-overlapping. The final submitted entry additionally contained three labels set by hand, which the task Terms prohibit; this was disclosed to the organisers and is documented in Section 7. The system-only score is reported throughout and no rank is claimed.
Team thylinao entered the Open and Strict tracks of the BAREC 2026 shared task on sentence-level Arabic readability with a deliberately plain system: a fourteen-member equal-weight ensemble of encoder fine-tunes, decoded to the 19-level scale by cutpoints interpolated halfway between validation-tuned thresholds and thresholds that match the blind score distribution to the training label prior. It scored 85.3 QWK on both tracks, first on both. This paper is about how those choices were measured. Three independent estimates place the noise floor of our offline evaluation between 0.09 and 0.44 QWK, as large as the margins that decided the final ranking. In three paired comparisons against the live board, only the smallest of the three offline estimates survived; the other two reversed in sign. We therefore proved that the Open and Strict tracks score identical gold, and made every decision worth a submission from a live board reading. Once the board showed the mean was capped, every candidate we probed came back a tie, bar one that we later withdrew as outside the rules. The full ledger, including four refuted hypotheses, is in the appendix.
This paper describes the first place system of the AKBC 2026 shared task on predicting complete knowledge base entries from language models, built on a single 31.3B parameter base model in completion mode with no training and no retrieval. It scores 0.7060 on the official test set; the runner up scores 0.6908, and a composite taking the best per relation score of every other ranked team would reach 0.7011.

The largest single improvement I found is not an algorithm. Asking the model to recall facts about an entity before committing to an answer, rather than asking for the answer directly, moved hasArea from 0.8100 to 0.8700. Three earlier configurations, across two frames and two aggregation methods, had all returned exactly 0.8100, which I had read as evidence that the answers were absent from the candidate pool. That reading was incorrect.

The main analytical result is a correction to how this field measures headroom, and it corrects my own earlier conclusion as much as anyone else’s. Oracle coverage analyses report that hasCapacity is selection bound: the recitation frame puts an acceptable answer in the pool for 84.5 percent of validation subjects while the aggregator realises 36.1 percent. That inference does not survive a null control. A pool of one hundred guesses spanning one decade contains about twenty one mutually exclusive 5 percent tolerance windows, so it contains something acceptably close to almost any plausible target whether or not the model knows the answer. Permuting gold values across subjects measures that coincidence rate directly: 0.355 for hasCapacity against 0.031 for hasArea. Chance corrected, the findable headroom beyond the rank one answer falls from 0.443 to 0.203. Three instruments that are not functions of vote frequency then agree that the relation is knowledge bound.