looking for contributors - trie based memory efficient LLM…
0 reactions
looking for contributors - trie based memory efficient LLM runner
SALT shrinks a long document down to a fixed size before it is sent to a language model, keeping the sentences that carry the most information. It works with any model, produces a shorter plain-text prompt, and cuts the compute, memory, and wait time that long inputs cost. saltChat keeps the theme trie in DRAM across turns, so a document is indexed once and reused for the whole conversation instead of being re-read every message.
#technology
SALT shrinks a long document down to a fixed size before it is sent to a language model, keeping the sentences that carry the most information. It works with any model, produces a shorter plain-text prompt, and cuts the compute, memory, and wait time that long inputs cost. saltChat keeps the theme trie in DRAM across turns, so a document is indexed once and reused for the whole conversation instead of being re-read every message.
#technology
| 3,000 mlx | |
| 4 | |
| 0 |