0 reactions
Large in-memory caches were causing GC pauses in our Node service, so I built an off-heap cache addon for it
If you've ever run a Node service with a big in-process cache (tens of thousands of entries, JSON blobs, that kind of thing) you've probably seen p99 latency spike during V8 garbage collection — the more live objects sit in the heap, the longer mark-sweep takes, and there's not much you can do about it from JS land since the GC doesn't know your cache entries are "just cache" and safe to deprioritize.
I built OffHeap to get around this: it's a cache that stores its data outside the V8 heap entirely (native memory managed from a small Rust layer via NAPI-RS), so the objects never show up in V8's GC graph at all. The JS-facing API is a normal cache — get/set/delete/TTL — with LRU, ARC, and W-TinyLFU eviction policies to choose from.
Under a synthetic GC-pressure test (500k keys, 1M ops), the worst single GC stop-the-world pause dropped from \~300ms (plain in-heap cache) to \~11ms. Average per-op latency is a bit higher than a pure in-heap Map (it's crossing an FFI boundary, that's not free), but the tail latency and memory behavior under load is the whole point.
It's on npm (\`offheap\`), dual-licensed MIT/Apache-2.0, docs at the repo. Full disclosure, this is my project — I actually shipped a broken cross-platform install for a bit (CI wasn't publishing the per-platform binaries correctly) and just fixed that, so if anyone tries it and hits install issues, please tell me. Genuinely looking for people to poke holes in it before I call it stable.
#dev #js #technology #programming
source
If you've ever run a Node service with a big in-process cache (tens of thousands of entries, JSON blobs, that kind of thing) you've probably seen p99 latency spike during V8 garbage collection — the more live objects sit in the heap, the longer mark-sweep takes, and there's not much you can do about it from JS land since the GC doesn't know your cache entries are "just cache" and safe to deprioritize.
I built OffHeap to get around this: it's a cache that stores its data outside the V8 heap entirely (native memory managed from a small Rust layer via NAPI-RS), so the objects never show up in V8's GC graph at all. The JS-facing API is a normal cache — get/set/delete/TTL — with LRU, ARC, and W-TinyLFU eviction policies to choose from.
Under a synthetic GC-pressure test (500k keys, 1M ops), the worst single GC stop-the-world pause dropped from \~300ms (plain in-heap cache) to \~11ms. Average per-op latency is a bit higher than a pure in-heap Map (it's crossing an FFI boundary, that's not free), but the tail latency and memory behavior under load is the whole point.
It's on npm (\`offheap\`), dual-licensed MIT/Apache-2.0, docs at the repo. Full disclosure, this is my project — I actually shipped a broken cross-platform install for a bit (CI wasn't publishing the per-platform binaries correctly) and just fixed that, so if anyone tries it and hits install issues, please tell me. Genuinely looking for people to poke holes in it before I call it stable.
#dev #js #technology #programming
source
| 1,000 mlx | |
| 2 | |
| 0 |
0 comments