Financial language agents are commonly evaluated through static prediction tasks or isolated historical backtests, where an agent’s decisions cannot affect subsequent prices or the behavior of competing strategies. This overlooks a defining property of real markets: interpretations of financial news influence order flow, price formation, and the competitive environment faced by other participants. We introduce FINEVO, a closed-loop evaluation framework that integrates time-stamped financial news, heterogeneous trading agents, a continuous double-auction market, and population-level capital reallocation. FINEVO evaluates language-based agents alongside technical, forecasting, reinforcement-learning, and informed strategies, reporting not only returns but also market concentration, strategic reorganization, regime-dependent standing, and recovery from shocks. Across 128-run suites of controlled market shocks and historical-news-conditioned episodes, we find that language-agent performance is strongly contingent on the surrounding market regime and agent population. Endogenous price formation also causes identical information schedules to produce different outcomes under different competitive compositions. Mechanism ablations further show that innovation is critical for maintaining contestability: removing it increases mean HHI from 0.189 to 0.477 and mean Top-1 capital share from 0.344 to 0.531. These results suggest that financial language agents should be evaluated not only by how accurately they interpret information, but also by how their decisions reshape the markets in which they operate.

In sectors in where data quality is critical, like finance and healthcare, it is crucial to have confidence in not only the outputs generated by retrieval-augmented generation (RAG) models but also the process followed by the model while arriving at the output. Existing methods, such as hallucination detection and input-output entailment measurements, fail to capture the model’s internal state during answer generation. This paper introduces a novel approach to predict the correctness of the generated answer by modeling the model’s uncertainty on quantified perturbations of input. Extensive experiments across multiple large language models (LLMs) demonstrate that our approach quantifies RAG robustness by aligning predictions with ground truth with a Avg.Mean Square Error (MSE) 0.002 while offering flexibility for diverse qualitative metrics.

“Deploying tuning-free prompting is challenging in engineering practice: it not only requiresusers to engage in cumbersome trials and errors but is also extremely time-consuming,as even a slight change in wording and phrasing could have a huge impact on the finalperformance. To further investigate the impact of different prompts, in this work, weperform a systematic inspection of four factors in linguistics involved in prompt engineering:syntax, semantics, lexicon, and pragmatics. The empirical results quantify the sensitivityof the output to small textual perturbations in four linguistic factors of prompts. Basedon the analysis of these four factors, we present a series of design guidelines to helphuman users write effective prompts. Human evaluation on amateurs shows that usingthe proposed guidelines helps humans produce prompts with significant gains in zero-shotperformance in Pre-trained Language Models (PLMs) and hence validates the utility ofthe guidelines.”

Emotional conversation systems generate responses for the input queries considering the speaker’s emotions in a conversation. Existing emotional conversation systems output emotional responses according to either a given emotion or the user’s emotion reflected in the input queries. Following a given emotion may lead to an emotional drift between the given emotion and the conversation state, and following only the user’s emotion may aggravate the user’s negative feelings if users suffer from a negative mood. In this paper, we propose to generate empathetic responses catering to the user’s emotions while leading the conversation to be emotionally positive. Particularly, by abstracting the conversation corpus, we extract and store the different responding strategies for different users’ emotions and conversational topics into a memory. We encourage positive emotions in conversation via a sentiment evaluator. We model the memory outputs with a Gaussian mixture distribution and sample a final responding strategy from the distribution. The strategy acts as a condition to a transformer model to generate responses. The experiments verify our model surpasses the baseline methods in appropriateness, diversity, and generating emotionally positive responses.