Compressed Convolutional Attention in Latent Space

From full-head freedom to shared latent bottlenecks

Each rung targets a different pain point. MQA and GQA mostly attack decode-time KV duplication. MLA compresses storage but often expands for computation. CCA compresses the computation workspace itself.

query diversity retained

shared KV structure

compressed latent workspace

Inside the compressed latent workspace

Use the stepper to move from baseline MHA to MLA to CCA. The left heatmap shows score computation shape. The right flow diagram shows where compression happens: cache-only vs compute-path.

Step reading

MHA computes and stores everything at full head width. MLA stores compactly but often re-expands for attention math. CCA keeps the interaction inside latent width. CCGQA stacks latent compute with head sharing.

Heatmap cue

Brighter cells represent heavier effective interaction cost. The visual mock data concentrates energy differently to illustrate how shared latent channels can reduce width without deleting token connectivity.

Practical question

The transcript’s core doubt remains: how much of the win is the new attention math itself versus a broader package of convolutional mixing, head grouping, and budget reallocation in MoE setups?

Performance explorer

These charts use realistic illustrative values drawn from the episode narrative: CCA-family methods aim to improve prefill throughput, cut backward cost, and keep quality competitive under equal or stricter cache budgets.

Long-context economics: where the bill lands

Hover the grid to inspect how prompt length shifts pressure between prefill compute and decode memory bandwidth. The bottom stream view shows why methods that only shrink KV storage solve a different slice of the problem than latent-space attention.