Fine-tuning large language models on domain data is a standard practice, yet vanilla pipelines rely on a costly data mixture sweep to avoid catastrophic forgetting of general capabilities. We propose Dynamic Anchor Annealing (DAA), a full-parameter fine-tuning framework that decouples general and domain learning at the gradient level rather than at the data level. DAA estimates a global gradient anchor once on general data through zero learning rate training, anchors the domain gradient toward this anchor during fine-tuning, modulates step sizes by the cosine similarity between the two gradients, and combines this with annealing learning. Under this design, fine-tuning relies solely on domain data and does not lose general capability. Across four domains and three base models, DAA improves the joint domain and general harmonic mean by 12.5 points on average over direct fine-tuning and by 3.8 points over vanilla data mixture fine-tuning, and the margin widens to 66.9 points on a temporal out-of-distribution setting where direct fine-tuning collapses. Our headline 91% reduction in GPU hours is a workflow level saving from removing the mixture ratio sweep rather than a per run speedup, since the per run cost of DAA is comparable to direct fine-tuning.

Large language models (LLMs) have recently demonstrated excellent performance in text embedding tasks. Previous work usually use LoRA to fine-tune existing LLMs, which are limited by the data and training gap between LLMs and embedding models. In this work, we introduce Conan-embedding-v2, a new 1.4B-parameter LLM trained from scratch and fine-tuned as a text embedder. First, we add news data and multilingual pairs for LLM pretraining to bridge the data gap. Based on this, we propose a cross-lingual retrieval dataset that enables the LLM to better integrate embeddings across different languages. Second, whereas LLMs use a causal mask with token-level loss, embedding models use a bidirectional mask with sentence-level loss. This training gap makes full fine-tuning less effective than LoRA. We introduce a soft-masking mechanism to gradually transition between these two types of masks, enabling the model to learn more comprehensive representations. Based on this, we propose a dynamic hard negative mining method that exposes the model to more difficult negative examples throughout the training process. Being intuitive and effective, with only approximately 1.4B parameters, Conan-embedding-v2 achieves SOTA performance on both the Massive Text Embedding Benchmark (MTEB) and Chinese MTEB (May 19, 2025).