Kohonen's 1972 Correlation Matrix Memory, Decades Before Attention
Teuvo Kohonen · Dept. of Technical Physics, Helsinki University of Technology |
AI Post Transformers podcast companion
How a Correlation Matrix Memory is built
Every stored pair (key q, data x) adds one outer product to a shared matrix M. Recall is a single matrix–vector multiply — no iteration, no gradient, no learning signal.
c·Σ xᵖqᵖᵀ
matrix construction rule
1
matrix-vector multiply to recall
0
gradients / backprop steps
Recall equation, expanded
Substituting the stored sum back into the recall step splits the result into the wanted signal plus interference from every other stored pattern.
The correlation matrix, cell by cell
Toggle key orthogonality. With orthogonal keys, recall is exact. Once keys correlate, cross-terms bleed noise into every stored pattern's recall — this is crosstalk.
low valuemid valuehigh value / crosstalk spike
Retina reconstruction — Fig. 2 replica
140-element retina, 4000 associators, two superimposed patterns. Near-orthogonal keys reconstruct cleanly; low-correlation keys come back noisier.
Where crosstalk comes from
Two stored key vectors overlap in the same associator cells. The recall of one pattern picks up a scaled contribution from the other.
CCMM vs ICMM: wiring completeness
A Complete Correlation Matrix Memory (CCMM) wires every key-to-data connection. An Incomplete one (ICMM) keeps only a random subset — this is the fault-tolerance claim.
Noise bound vs number of associators (Example 1)
Relative standard deviation of recall shrinks toward zero as associator count s grows — a law-of-large-numbers curve. The paper's worked bound: s ≳ 100·n keeps relative std dev under 0.1, independent of key size m.
relative std deviation0.1 target threshold
Fifty years of outer-product memory
Hover a node for what it contributed. The dashed edge marks the disputed link — structural resemblance to attention, no shared learning mechanism.
Claim vs validation level
The noise-bound math is rigorous but only checked once, qualitatively, at P=2 on one retina. The pruning/unsupervised-learning claim and the classifier claim go further than what was tested.