PsychMet is a domain-grounded chatbot that uses a GPT-4.1 conversational model with retrieval-augmented generation over a curated psychometrics corpus, with emphasis on IRT and NCME competencies. Using the RAGAS framework on a 30-question set, PsychMet achieved an Overall score of 0.539, with strengths in Answer Correctness (0.810) and Context Recall (0.671), moderate Faithfulness (0.588), and weaknesses in Answer Relevancy (0.284), Context Precision (0.425), and Context Relevancy (0.474). This pattern suggests that retrieval breadth is outpacing specificity. We outline targeted fixes — such as hybrid sparse+dense retrieval with light filtering and question-first prompting — to tighten focus without sacrificing coverage. PsychMet is accurate and transparently sourced for exploratory learning; with retrieval tightening and answer scoping, it can better support time-bound professional workflows.

Field testing is a resource-intensive bottleneck in test development. This study applied an interpretable framework that leverages a Large Language Model (LLM) for structured feature extraction from TIMSS items. These features will train several classifiers, whose predictions will be explained using SHAP, providing actionable, diagnostic insights insights for item writers.