I trained a model and it learned gradient descent. So I…
0 reactions
I trained a model and it learned gradient descent. So I deleted the trained part, accuracy stayed the same.
Built a system for NLI where instead of `h → Linear → logits`, the hidden state evolves over a few steps before classification. Three learned anchor vectors define basins (entailment / contradiction / neutral), and the state moves toward whichever basin fits the input.
The surprising part came after training.
**The learned update collapsed to a closed-form equation**
The update rule was a small MLP — trained end-to-end on \~550k examples. After systematic ablation, I found the trained dynamics were well-approximated by a simple energy function:
V(h) = −log Σ exp(β · cos(h, Aₖ))
Replacing the entire trained MLP with the analytical gradient:
h_{t+1} = h_t − α∇V(h_t)
→ same accuracy.
The claim isn't that the equation is surprising in hindsight. It's that I didn't design it — I trained a black-box MLP and found afterward that it had converged to this. And I could verify it by deleting the MLP entirely. The surprise isn't the equation, it's that the equation was recoverable at all.
**Three observed patterns (not laws — empirical findings)**
1. **Relational initialization** — `h₀ = v_hypothesis − v_premise` works as initialization without any learned projection. This is a design choice, not a discovery — other relational encodings should work too.
2. **Energy structure** — the representation space behaves like a log-sum-exp energy over anchor cosine similarities. Found empirically.
3. **Dynamics** (the actual finding) — inference corresponds to gradient descent on that energy. Found by ablation: remove the MLP, substitute the closed-form gradient, nothing breaks.
Each piece individually is unsurprising. What's worth noting is that a trained system converged to all three without being told to — and that convergence is verifiable by deletion, not just observation.
**Failure mode: universal fixed point**
Trajectory analysis shows that after \~3 steps, most inputs collapse to the same attractor state regardless of input. This is a useful diagnostic: it explains exactly why neutral recall was stuck at \~70% — the dynamics erase input-specific information before classification. Joint retraining with an anchor alignment loss pushed neutral recall to 76.6%.
The fixed point finding is probably the most practically useful part for anyone debugging class imbalance in contrastive setups.
**Numbers (SNLI, BERT encoder)**
| | Old post | Now |
|---|---|---|
| Accuracy | 76% (mean pool) | 82.8% (BERT) |
| Neutral recall | 72.2% | 76.6% |
| Grad-V vs trained MLP | — | accuracy unchanged |
The accuracy jump is mostly the encoder (mean pool → BERT), not the dynamics — the dynamics story is in the neutral recall and the last row.
📄 Paper: https://zenodo.org/records/19092511
💻 Code: https://github.com/chetanxpatil/livnium
**Still need an arXiv endorsement** ( cs.CL or cs.LG) — this will be my first paper. Code: **HJBCOM** → https://arxiv.org/auth/endorse
Feedback welcome, especially on pattern 1 — I know it's the weakest of the three.
#technology
source
Built a system for NLI where instead of `h → Linear → logits`, the hidden state evolves over a few steps before classification. Three learned anchor vectors define basins (entailment / contradiction / neutral), and the state moves toward whichever basin fits the input.
The surprising part came after training.
**The learned update collapsed to a closed-form equation**
The update rule was a small MLP — trained end-to-end on \~550k examples. After systematic ablation, I found the trained dynamics were well-approximated by a simple energy function:
V(h) = −log Σ exp(β · cos(h, Aₖ))
Replacing the entire trained MLP with the analytical gradient:
h_{t+1} = h_t − α∇V(h_t)
→ same accuracy.
The claim isn't that the equation is surprising in hindsight. It's that I didn't design it — I trained a black-box MLP and found afterward that it had converged to this. And I could verify it by deleting the MLP entirely. The surprise isn't the equation, it's that the equation was recoverable at all.
**Three observed patterns (not laws — empirical findings)**
1. **Relational initialization** — `h₀ = v_hypothesis − v_premise` works as initialization without any learned projection. This is a design choice, not a discovery — other relational encodings should work too.
2. **Energy structure** — the representation space behaves like a log-sum-exp energy over anchor cosine similarities. Found empirically.
3. **Dynamics** (the actual finding) — inference corresponds to gradient descent on that energy. Found by ablation: remove the MLP, substitute the closed-form gradient, nothing breaks.
Each piece individually is unsurprising. What's worth noting is that a trained system converged to all three without being told to — and that convergence is verifiable by deletion, not just observation.
**Failure mode: universal fixed point**
Trajectory analysis shows that after \~3 steps, most inputs collapse to the same attractor state regardless of input. This is a useful diagnostic: it explains exactly why neutral recall was stuck at \~70% — the dynamics erase input-specific information before classification. Joint retraining with an anchor alignment loss pushed neutral recall to 76.6%.
The fixed point finding is probably the most practically useful part for anyone debugging class imbalance in contrastive setups.
**Numbers (SNLI, BERT encoder)**
| | Old post | Now |
|---|---|---|
| Accuracy | 76% (mean pool) | 82.8% (BERT) |
| Neutral recall | 72.2% | 76.6% |
| Grad-V vs trained MLP | — | accuracy unchanged |
The accuracy jump is mostly the encoder (mean pool → BERT), not the dynamics — the dynamics story is in the neutral recall and the last row.
📄 Paper: https://zenodo.org/records/19092511
💻 Code: https://github.com/chetanxpatil/livnium
**Still need an arXiv endorsement** ( cs.CL or cs.LG) — this will be my first paper. Code: **HJBCOM** → https://arxiv.org/auth/endorse
Feedback welcome, especially on pattern 1 — I know it's the weakest of the three.
#technology
source
| 3,000 mlx | |
| 5 | |
| 0 |