Real-world deployments of large language models (LLMs) increasingly rely on user-provided instructions that are long, precise, and tailored to specific workflows. In such settings, aggregate leaderboard performance is often a weak proxy for a model’s ability to faithfully execute the particular instruction that a practitioner intends to deploy. To address this gap, we introduce FlexBench/FlexEval, an instruction-conditioned evaluation framework that instantiates a specialized test suite from a user-provided target instruction. Given a target instruction, FlexBench (i) derives a set of verifiable evaluation dimensions and (ii) generates an instruction-conditioned conversation corpus. FlexEval then aggregates dimension-level judgments into two complementary metrics: Coverage, which measures workflow progress, and Achievement, which measures conditional correctness. We evaluate the framework on nine multi-turn target instructions spanning diverse scenario categories and workflow topologies, across nine contemporary LLMs over 3,888 generated dialogues. The results show that model rankings and failure modes vary substantially across target instructions, highlighting the need for instruction-conditioned evaluation beyond fixed aggregate benchmarks. Our source code is publicly available at https://github.com/Coryyy/FlexBench.

Parameter-efficient fine-tuning (PEFT) is a popular method for tailoring pre-trained large language models (LLMs), especially as the models’ scale and the diversity of tasks increase. Low-rank adaptation (LoRA) is based on the idea that the adaptation process is intrinsically low-dimensional, i.e., significant model changes can be represented with relatively few parameters. However, decreasing the rank encounters challenges with generalization errors for specific tasks when compared to full-parameter fine-tuning. We present MELoRA, a mini-ensemble low-rank adapters that uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential.The core idea is to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters. This can capture a significant degree of diversity among mini LoRAs, thus promoting better generalization ability. We conduct a theoretical analysis and empirical studies on various NLP tasks. Our experimental results show that, compared to LoRA, MELoRA achieves better performance with 8 times fewer trainable parameters on natural language understanding tasks and 36 times fewer trainable parameters on instruction following tasks, which demonstrates the effectiveness of MELoRA.