To mitigate high inference costs for in-context learning, prior works have proposed activation-space task vectors that encapsulate demonstration contexts into static representations. However, performance occasionally degrades across tasks, leaving it unclear whether this stems from sub-optimal extraction heuristics or fundamental limitations. We resolve this by decomposing Transformer self-attention to establish a theoretical best-case approximation: the Minimal-Gap Approximation. By isolating structural constraints from heuristic flaws, the Minimal-Gap Approximation reveals that task vectors are fundamentally bottlenecked by attention and length gaps. Specifically, because task vectors lock contextual information to fixed attention maps and positional alignments, they structurally fail under dynamic routing, length mismatches, and multi-token generation. We further show that these findings consistently hold across training-free extract-and-inject task vector methods, model architectures, and model scales.