Adaptive compression for inverted index Adaptive…
0 reactions
Adaptive compression for inverted index
Adaptive compression is critically important for analytical workloads as it reduces storage costs and can dramatically speedup query execution. Great reads on this: DuckDB lightweight compression https://duckdb.org/2022/10/28/lightweight-compression , BtrBlocks https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/papers/btrblocks.pdf and FastLanes https://ir.cwi.nl/pub/35881/35881.pdf .
Inverted indexes deal with millions of sorted integers and could benefit from the same ideas, but posting lists have their own specifics: values are monotonically increasing integers, blocks are much smaller compared to columnar storage and the access pattern is typically different.
The article is a deep dive into how we implemented per-block adaptive compression for our open-source inverted indexing engine IResearch, what problems we ran into and how we solved them.
GitHub: https://github.com/serenedb/serenedb
#programming #technology #dev
Adaptive compression is critically important for analytical workloads as it reduces storage costs and can dramatically speedup query execution. Great reads on this: DuckDB lightweight compression https://duckdb.org/2022/10/28/lightweight-compression , BtrBlocks https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/papers/btrblocks.pdf and FastLanes https://ir.cwi.nl/pub/35881/35881.pdf .
Inverted indexes deal with millions of sorted integers and could benefit from the same ideas, but posting lists have their own specifics: values are monotonically increasing integers, blocks are much smaller compared to columnar storage and the access pattern is typically different.
The article is a deep dive into how we implemented per-block adaptive compression for our open-source inverted indexing engine IResearch, what problems we ran into and how we solved them.
GitHub: https://github.com/serenedb/serenedb
#programming #technology #dev
| 2,000 mlx | |
| 4 | |
| 0 |