Standard transformers use pairwise (2D) self-attention, which models only token-to-token interactions. Sparse attention reduces computational cost while keeping the same pairwise structure and cannot capture richer multi-token interactions. Therefore, we propose Higher-Order Block Attention (HOBA), which models explicit triplet interactions using structured 3D attention with block-diagonal unrolling. This design remains computationally efficient while extending attention beyond pairwise relations. Additionally, we demonstrate that cross-block interaction improves HOBA performance by supporting long-range token dependencies beyond triplet modeling. Experiments on ten NLP benchmarks across five tasks show that HOBA consistently outperforms dense and sparse 2D baselines.

Analog in-memory computing (AIMC) offers substantial efficiency gains for transformer inference but introduces hardware-induced noise that can distort attention behavior. Prior studies primarily focus on AIMC evaluations for vision tasks and CNN-based models. They largely overlook how hardware-induced noise perturbs internal attention dynamics in NLP models. In this work, we present the first fine-grained analysis of analog vulnerability in pretrained transformers, examining projection submodules, attention heads, and layer-wise dynamics across multiple NLP tasks. Results show that query (Q), key (K), and value (V) projections are the most sensitive components, while feed-forward layers remain comparatively robust. Also, analog noise yields depth-dependent degradation in higher layers, leading to scattered attention and disrupted token routing. This pre-deployment analysis mitigates potential resource misuse before physical deployment and offers practical guidance for designing noise-resilient analog NLP transformers.