SubmissionNumber#=%=#31
FinalPaperTitle#=%=#Controlled Fine-Tuning for Two-Tower Recommenders: Why Gradual Unfreezing Can Fail and Cosine Regularization Suffices
ShortPaperTitle#=%=#
NumberOfPages#=%=#5
CopyrightSigned#=%=#Djallel Bouneffouf
JobTitle#==#
Organization#==#
Abstract#==#Controlled fine-tuning of pretrained text
encoders—via gradual layer-wise unfreezing
and anchoring—is widely assumed to bene-
fit downstream tasks. We test this assump-
tion in two-tower recommendation and find
three negative results. First, any fine-tuning
dramatically beats a frozen encoder (≥ 2.7×
cold Recall@10 lift); freezing the item tower
is strongly counterproductive. Second, gradual
unfreezing worsens performance compared to
vanilla fine-tuning under realistic budgets (12
epochs). Third, a fixed cosine drift penalty is
the only consistently beneficial component; our
full adaptive recipe does not outperform vanilla
fine-tuning. Thus, for two-tower retrieval, prac-
titioners should default to vanilla fine-tuning
with light regularization—gradual unfreezing
is unnecessary and harmful, challenging con-
ventional wisdom from language model fine-
tuning.
Author{1}{Firstname}#=%=#Gabriel
Author{1}{Lastname}#=%=#Dershowitz
Author{1}{Orcid}#=%=#
Author{1}{Email}#=%=#gdershow@mail.yu.edu
Author{1}{Affiliation}#=%=#YU
Author{2}{Firstname}#=%=#Djallel
Author{2}{Lastname}#=%=#Bouneffouf
Author{2}{Username}#=%=#djallel_info
Author{2}{Orcid}#=%=#
Author{2}{Email}#=%=#djallel_info@yahoo.fr
Author{2}{Affiliation}#=%=#IBM

==========
èéáğö