GitHub - AyushSuri8/nexus-search-engine: Distributed…
0 reactions
GitHub - AyushSuri8/nexus-search-engine: Distributed search engine implementing BM25, HNSW vector search, LSM storage, Bloom filters, and W-TinyLFU caching.
Modern search engines combine multiple retrieval techniques:
lexical search (BM25), semantic vector search, caching, and ranking.
I wanted to understand how these components interact,
so I implemented a miniature search pipeline from scratch.
Key parts:
• Bloom filter to skip zero-result queries
• LSM-tree backed inverted index
• HNSW graph for semantic vector search
• W-TinyLFU admission-aware caching
• Reciprocal Rank Fusion to merge rankings
One interesting optimization was using skip pointers in the
posting lists to reduce intersection complexity from O(n*m)
to roughly O(n * sqrt(m)).
Another was using deterministic N-gram embeddings to avoid
external embedding APIs.
Full writeup + code:
https://github.com/AyushSuri8/nexus-search-engine
#technology
Modern search engines combine multiple retrieval techniques:
lexical search (BM25), semantic vector search, caching, and ranking.
I wanted to understand how these components interact,
so I implemented a miniature search pipeline from scratch.
Key parts:
• Bloom filter to skip zero-result queries
• LSM-tree backed inverted index
• HNSW graph for semantic vector search
• W-TinyLFU admission-aware caching
• Reciprocal Rank Fusion to merge rankings
One interesting optimization was using skip pointers in the
posting lists to reduce intersection complexity from O(n*m)
to roughly O(n * sqrt(m)).
Another was using deterministic N-gram embeddings to avoid
external embedding APIs.
Full writeup + code:
https://github.com/AyushSuri8/nexus-search-engine
#technology
| 3,000 mlx | |
| 4 | |
| 0 |