I open-sourced TRACER: replace 91% of LLM classification…
0 reactions
I open-sourced TRACER: replace 91% of LLM classification calls with a llightweigth ML surrogate trained on your LLM's own outputs
If you're running an LLM for classification, 91% of your traffic is probably simple
enough for a surrogate model trained on your LLM's own outputs.
TRACER learns which inputs it can handle safely - with a formal guarantee it'll
agree with the LLM at your target rate. If it can't clear the bar, it doesn't deploy.
pip install tracer-llm && tracer demo
HN: https://news.ycombinator.com/item?id=47573212
#technology
If you're running an LLM for classification, 91% of your traffic is probably simple
enough for a surrogate model trained on your LLM's own outputs.
TRACER learns which inputs it can handle safely - with a formal guarantee it'll
agree with the LLM at your target rate. If it can't clear the bar, it doesn't deploy.
pip install tracer-llm && tracer demo
HN: https://news.ycombinator.com/item?id=47573212
#technology
| 1,000 mlx | |
| 3 | |
| 0 |