the theoretical ceiling of purely autoregressive models…
0 reactions
the theoretical ceiling of purely autoregressive models
Are we basically trying to emulate deterministic search with probabilistic brute-force right now?
been thinking about how weird the current ai paradigm is from a pure cs theory standpoint. we spent decades building robust constraint satisfaction algorithms and formal verification methods. then transformers blew up, and suddenly the entire industry is trying to force a next-token probability engine to do strict, multi-step logic.
it just feels mathematically ineFficient. no matter how much compute you throw at a transformer, it's still fundamentally a probability distribution over a discrete vocabulary. It can't natively backtrack or satisfy global constraints, it just guesses forward
I've noticed some pushback against this recently, with some research pivoting back to continuous mathematical spaces. for instance, looking at how Logical Intelligence https://logicalintelligence.com/ uses energy-based models to treat logic as a pure constraint satisfaction problem rather than a token generation one. Fnding a low-energy state that respects all constraints just aligns so much better with traditional computer science principles
it honestly feels like we temporarily ignored fundamental cs theory just because scaling huge probability matrices was easier in the short term. It’ll be interesting to see if the industry hits a hard theoretical wall with transformers soon.
#technology
Are we basically trying to emulate deterministic search with probabilistic brute-force right now?
been thinking about how weird the current ai paradigm is from a pure cs theory standpoint. we spent decades building robust constraint satisfaction algorithms and formal verification methods. then transformers blew up, and suddenly the entire industry is trying to force a next-token probability engine to do strict, multi-step logic.
it just feels mathematically ineFficient. no matter how much compute you throw at a transformer, it's still fundamentally a probability distribution over a discrete vocabulary. It can't natively backtrack or satisfy global constraints, it just guesses forward
I've noticed some pushback against this recently, with some research pivoting back to continuous mathematical spaces. for instance, looking at how Logical Intelligence https://logicalintelligence.com/ uses energy-based models to treat logic as a pure constraint satisfaction problem rather than a token generation one. Fnding a low-energy state that respects all constraints just aligns so much better with traditional computer science principles
it honestly feels like we temporarily ignored fundamental cs theory just because scaling huge probability matrices was easier in the short term. It’ll be interesting to see if the industry hits a hard theoretical wall with transformers soon.
#technology
| 4,000 mlx | |
| 4 | |
| 0 |