Xiaolongwang

Author directory

2026

Scientific research is not merely a sequence of tasks to be automated, but a knowledge-production process in which claims must remain verifiable, criticizable, revisable, and accountable. Yet most AutoResearch systems remain single scaffolds that collapse generation, execution, interpretation, and review into one loop. This survey diagnoses three resulting epistemic limits of such architectures: collapsed skepticism, where a single scaffold produces not only errors but also the narratives that allow them to survive; unverified translation from discovery to application, where proposal generation outpaces independent verification; and invisible evidence trails, where final manuscripts hide the failed attempts and post-hoc decisions that produced them. To overcome these structural fragilities, we propose a shift from autonomous AI scientists to AI research communities, where specialized agents, shared memory, inspectable artifacts, and governance checkpoints distribute epistemic labor. We synthesize AutoResearch as a transition from metric-driven search loops to AI-scientist workflows, domain-mediated discovery, research infrastructure, and accountable human–AI co-evolution. Across these layers, we argue for a fundamental shift from productivity-centered evaluation to verifiability-centered evaluation.
This paper argues that evaluation of educational LLMs should move beyond decontextualized task-based benchmarks toward situated benchmarks centered on pedagogical action. Prior work in human-computer interaction and the learning sciences has long emphasized situated use, user context, and interaction processes; our focus is narrower and complementary: how these concerns should reshape benchmark design for educational LLMs. Existing evaluations already cover question generation, knowledge answering, error diagnosis, feedback generation, and teaching analysis, but their dominant unit remains the single input-output episode, making it difficult to compare whether a model genuinely supports learning progress, instructional judgment, or subsequent revision in context. Educational activity is processual, context-dependent, and development-oriented, so explanations, feedback, and hints can only be evaluated properly in relation to learner state, interaction history, and pedagogical goals. We review a landscape still dominated by task-based paradigms and argue, from educational conceptions, educational values, and research methodology, that next-generation educational benchmarks should make pedagogical function in context a core target. We further propose layered, scalable, and benchmarkable situated protocols built through simulation-first development and follow-up empirical calibration, extending evaluation beyond task quality to contextual fit and downstream consequences for learning and teaching decisions.