LLM judges have emerged as a key approach to evaluating open-ended scientific outputs that resist automated verification, yet their faithfulness in reference-free settings remains underexplored, particularly for fields such as novel materials synthesis where no ground-truth reference exists. Such judges produce false positives that accept scientifically invalid recipes and false negatives that penalize semantically equivalent ones in unfamiliar notation. We introduce ASSAY, a probe suite built on domain-grounded augmentations of ground-truth synthesis recipes, and reveal that these two failure modes form a systematic Stability–Validity trade-off across thirteen judge models. To resolve this trade-off, we train SynthJudge-8B via DPO on contrastive pairs derived from the same augmentations, improving judgment accuracy from 51.4% to 85.1% while maintaining agreement with domain experts. We release our probe suite and trained model to support faithful automated evaluation in scientific fields beyond materials synthesis at https://github.com/SoyeonHH/ASSAY.