Column Type Annotation (CTA) enriches tables with semantic column types that are often missing or incomplete in real-world data, aiding multiple downstream table tasks. Yet CTA remains challenging because sparse or noisy column values often require table-level context for disambiguation. Prior work either flattens tables into encoder inputs, losing structure and semantics, or follows a description-then-encoding pipeline that separates encoding and adds generation cost. We propose D2R, a decoder-based framework that obtains column representations directly from the hidden state that initiates column description generation. Specifically, D2R first encodes the table context once as a shared prefix KV cache, providing table-level context for all target columns. For each column, we append its sampled values and a description trigger token, then extract the trigger’s hidden state as the column embedding, preserving semantics without producing text. A decoder-attached classifier, trained via contrastive learning aligned with the CTA objective, predicts column types directly from this embedding. Across four CTA benchmarks, D2R achieves the best Micro-F1, outperforming TURL, a strong table PLM, by up to 12.57 points, while retaining efficient inference. The code is available at https://github.com/IvesCheung/cta.