CoCA: Fusing Position Embedding with Collinear Constrained Attention in Transformers for Long Context Window Extending

CoCA: Fusing Position Embedding with Collinear Constrained Attention in Transformers for Long Context Window Extending Shiyi Zhu author Jing Ye author Wei Jiang author Siqiao Xue author Qi Zhang author Yifan Wu author Jianguo Li author 2024-08 text Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) Lun-Wei Ku editor Andre Martins editor Vivek Srikumar editor Association for Computational Linguistics Bangkok, Thailand conference publication zhu-etal-2024-coca 10.18653/v1/2024.acl-long.233 https://aclanthology.org/2024.acl-long.233/ 2024-08 4247 4262