Logical reasoning is a core ability of modern language models, and various approaches have been proposed to enhance it. One effective method is neuro-symbolic approach, which makes a model output logic formulas and predict the answer based on them. However, fair comparison is challenging because existing methods use different experimental settings, such as post-processing and prompt styles. In addition, existing error analyses are too coarse to capture the impact of sub-formula level structures. In this paper, we present a systematic analysis of neuro-symbolic approaches under a unified setting with sub-formula level annotation. We reveal that some sub-formula level structures bias the model’s inference toward assuming the answer is true, and neuro-symbolic approaches with an external inference tool help mitigate this bias. On the other hand, these methods sometimes lack flexibility in representing sentence meaning as logic formulas.

As language models continue to scale, the demand for knowledge editing, a retraining-free knowledge update method, has increased. However, since knowledge editing directly alters token prediction probabilities acquired during pretraining, the probabilities may diverge from the empirical distribution. In this study, we analyze the impact of knowledge editing to compare the alignment between token prediction probabilities and task accuracy by calculating confidence calibration before and after knowledge editing. Our results reveal that, for tasks requiring semantic understanding, the range of increase in token prediction probabilities tends to be smaller than that of accuracy improvement, suggesting that knowledge editing methods lead to less confidence in prediction.