Reference-free evaluation of LLM-generated code is essential when execution-based testing is unavailable or costly. We compare two paradigms: explicit LLM-as-a-Judge scoring, which assigns a quality score to a solution, and log-probability scoring, which uses log P𝜃(code ∣ task) as an instruction-free signal.Across HumanEval-X, we find that the two approaches capture qualitatively different aspects of code correctness. Explicit judges — particularly larger models — perform strongly on generated code, reflecting their ability to reason about task-solution alignment, but fail to distinguish correct solutions from minimally mutated ones. Log-probability exhibits the opposite pattern: weaker performance on generated code, but consistent pairwise separation of canonical from mutated solutions.These results reveal a discrimination-ranking dissociation and show that the two paradigms provide complementary, non-interchangeable signals: explicit judges capture semantic correctness, while log-probability captures local structural consistency.

We present MAGIC, a six-dimensional framework for extracting structured knowledge bases of hero and magic entity interactions from narrative texts of folktales and fairytales: each interaction between a story’s protagonist and a magical helper, villain, or object is decomposed along trigger modality, actor profile and agency balance, interaction mechanics, outcome valence, narrative function, and narrative position. We build an LLM-based extraction pipeline and apply it to a corpus of 1306 folktales collected from around the world, then show the schema supports rich structural analysis: unsupervised clustering over the extracted features recovers five recurring, readily nameable interaction archetypes – invoked and unsolicited helpers, a morally tested gatekeeper, a hero-controlled magic object, and an adversarial obstacle – that are stable across the three full-corpus extraction models (0.95 mean cluster-centroid similarity) and cut across both tale type and regional tradition rather than reproducing existing typological catalogues. We further characterize the schema’s reliability under LLM extraction (inter-model, not against a human gold standard): parse success and schema-conformance rates, a decomposition of extraction drift into field-confusion versus vocabulary-incompleteness failure modes, and cross-model field-level agreement, which varies substantially across the six dimensions and is sensitive to how many tales are compared across models. Despite this field-level variability, the archetype-level structure remains stable, suggesting that knowledge bases built from LLM extraction should be evaluated at the level of the downstream structure they support.

The paper presents early results in the development of an approach to predictive modeling of human developer perceiving of code generated in question-answering scenarios with Large Language Model (LLM) applications. The study is focused on building a model for the description and prediction of human implicit behavior during evaluative judgment of generated code through evaluation of its consistency, correctness, and usefulness as subjective perceiving characteristics. We used Markov Decision Process (MDP) as a basic framework to describe the human developer and his/her perceiving. We consider two approaches (regression-based and classification-based) to identify MDP parameters so it can be used as an “artificial” developer for human-centered code evaluation. An experimental evaluation of the proposed approach was performed with survey data previously collected for several code generation LLMs in a question-answering scenario. The results show overall good performance of the proposed model in acceptance rate prediction (accuracy 0.82) and give promising perspectives for further development and application.

Within this study, we evaluated a series of code generation models based on CodeGen and GPTNeo to compare the metric-based performance and human evaluation. For a deeper analysis of human perceiving within the evaluation procedure we’ve implemented a 5-level Likert scale assessment of the model output using a perceiving model based on the Theory of Planned Behavior (TPB). Through such analysis, we showed an extension of model assessment as well as a deeper understanding of the quality and applicability of generated code for practical question answering. The approach was evaluated with several model settings in order to assess diversity in quality and style of answer. With the TPB-based model, we showed a different level of perceiving the model result, namely personal understanding, agreement level, and readiness to use the particular code. With such analysis, we investigate a series of issues in code generation as natural language generation (NLG) problems observed in a practical context of programming question-answering with code.