The idea in one paragraph: over 2x2 real matrices, i² = −1 has two orthogonal solutions, rotation by +90 and −90 degrees:
i₊ = [[0, −1], [1, 0]], i₋ = [[0, 1], [−1, 0]]
Square either one and you get [[−1, 0], [0, −1]] = −I, so both are genuine square roots of minus one. Their combination is
S(θ) = cosθ · i₊ + sinθ · i₋ = [[0, −(cosθ − sinθ)], [cosθ − sinθ, 0]]
whose off-diagonal entry is
±(cosθ − sinθ)
That ± in front is a binary object sitting inside a smooth formula. SANATAN promotes it into an explicit soft variable:
r = tanh( γ ( cos(θ+ω) − sin(θ+ω) ) ) ≈ ±1
Each token projection u becomes an angle θ = π·u / (1 + |u|), every feature dimension softly picks a root, and the six-channel feature map
Φ(x) = [ 1, cosθ, sinθ, r, r·cosθ, r·sinθ ]
makes the attention score factorize exactly:
⟨Φ(q), Φ(k)⟩ = (1 + r_q r_k)(1 + cos Δθ)
a per-dimension parity gate (0 or 2 for hard roots) times an angular similarity. The GIF above walks through it step by step.
Because the score is an inner product, the p=1 case runs as an exact recurrence with a constant-size state, no KV cache and no decay:
S_t = S_{t−1} + Φ(k_t) v_tᵀ, z_t = z_{t−1} + Φ(k_t), out = ⟨Φ(q), S⟩ / ⟨Φ(q), z⟩
What I measured (184M params, single seeded held-out protocol):
- The recurrence is exact in practice, not just on paper: parallel and recurrent computation of the trained model agree within 1e-4 nats (float32). There is a demo notebook that reproduces this on CPU in about two minutes on a small model (agreement ~4e-6 per logit).
- Quality survives the constant state: 3.951 held-out loss recurrent vs 3.966 for its full parallel parent, via a two-stage recipe (sharpened pretraining, brief p=1 adaptation). Deploying p>1-trained models through the recurrence costs ~5 nats, so that path is dead and the paper says so.
- Decode: fixed 14.4 MB state vs a KV cache that hits 2.4 GB at 32K context; 0.153 ms/token flat vs 0.49 to 7.3 ms/token for softmax decoding (3.2x to 48x from 1K to 32K).
- Ablation: the root-parity gate is worth +0.53 nats over the same angular kernel without it.
Paper (PDF): https://github.com/bhargavpatel431997/sanatan-attention/blob/main/paper/sanatan_v2.pdf
Code + runnable demo notebook: https://github.com/bhargavpatel431997/sanatan-attention
Visual explainer of the idea: https://github.com/bhargavpatel431997/sanatan-attention/blob/main/docs/MOTIVATION.md
Independent researcher, single GPU/TPU budget, so critiques and pointers are genuinely welcome, especially from anyone who has tried interaction-level gating inside linear attention before.
Thanks for reading,
Bhargav Patel
https://www.linkedin.com/in/bhargav-patel-63bb27121/
