Autoregressive training requires that the computation at each position ignore future tokens. Causal masking enforces this for self-attention, but efficient training recipes can still break causality when they compute shared statistics across the sequence dimension. We study two such operations: structured activation sparsity and block-scaled activation quantization. Applying these operators along the sequence dimension is attractive for hardware efficiency and can look harmless, but it can break causality when the mask or scale applied to a past token depends on future activations. We show that models do exploit this dependency: the training objective continues to improve under the non-causal operator, but causal validation degrades when the future-dependent computation is removed. We analyze why mitigations that keep the operator in the forward pass can postpone or weaken exploitation, but cannot remove the future-to-past dependency. The solution is to remove non-causal statistics from the forward computation used for prediction, and we show that existing sparsity and quantization recipes do respect this rule, though causality was not an explicit design decision.

Decoder-only large language models typically rely solely on masked causal attention, which limits their expressiveness by restricting information flow to one direction. We propose Bitune, a method that enhances pretrained decoder-only LLMs by incorporating bidirectional attention into prompt processing. We evaluate Bitune in instruction-tuning and question-answering settings, showing significant improvements in performance on commonsense reasoning, arithmetic, and language understanding tasks. Furthermore, extensive ablation studies validate the role of each component of the method, and demonstrate that Bitune is compatible with various parameter-efficient finetuning techniques and full model finetuning.

Transformer-based architectures have become the de-facto standard models for a wide range of Natural Language Processing tasks. However, their memory footprint and high latency are prohibitive for efficient deployment and inference on resource-limited devices. In this work, we explore quantization for transformers. We show that transformers have unique quantization challenges – namely, high dynamic activation ranges that are difficult to represent with a low bit fixed-point format. We establish that these activations contain structured outliers in the residual connections that encourage specific attention patterns, such as attending to the special separator token. To combat these challenges, we present three solutions based on post-training quantization and quantization-aware training, each with a different set of compromises for accuracy, model size, and ease of use. In particular, we introduce a novel quantization scheme – per-embedding-group quantization. We demonstrate the effectiveness of our methods on the GLUE benchmark using BERT, establishing state-of-the-art results for post-training quantization. Finally, we show that transformer weights and embeddings can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss. Our source code is available at https://github.com/qualcomm-ai-research/transformer-quantization.