Autoregressive training requires that the computation at each position ignore future tokens. Causal masking enforces this for self-attention, but efficient training recipes can still break causality when they compute shared statistics across the sequence dimension. We study two such operations: structured activation sparsity and block-scaled activation quantization. Applying these operators along the sequence dimension is attractive for hardware efficiency and can look harmless, but it can break causality when the mask or scale applied to a past token depends on future activations. We show that models do exploit this dependency: the training objective continues to improve under the non-causal operator, but causal validation degrades when the future-dependent computation is removed. We analyze why mitigations that keep the operator in the forward pass can postpone or weaken exploitation, but cannot remove the future-to-past dependency. The solution is to remove non-causal statistics from the forward computation used for prediction, and we show that existing sparsity and quantization recipes do respect this rule, though causality was not an explicit design decision.

Along with the performance improvement in NLP domain, the sizes of transformer-based language models (TLM) are also dramatically increased. Some prior works intend to compress TLM models into more compact forms, but do not fully consider the hardware characters may not support the efficient execution for these forms, leading to the deployment of TLM on hardware with noticeable acceleration is still challenging. This paper thoroughly designs a compression scheme named GPUSQ-TLM to maximally utilize the GPU-friendly 2:4 fine-grained structured sparsity and quantization characters. Especially, a dense TLM model is first pruned to meet the GPU’s acceleration constraint of sparse patterns with FP16 type, then it is further quantized into a fixed-point one by quantization-aware training, to provide an extra speedup for integer tensors on GPU. A mixed-strategy knowledge distillation of labels, logits and feature maps is used for best accuracy compensation during pruning and quantization process. Experiment results show GPUSQ-TLM scheme achieves state-of-the-art compression on TLM model of various encoder and decoder blocks with negligible accuracy degradation on SQuAD, GLUE, CNN-DM & XSum and WikiText benchmarking tasks. Moreover, GPUSQ-TLM can boost actual deployment performance by up to 4.08-4.25x latency and 6.18-6.79x throughput on A100 GPU.