We deleted our vector index
By Igor C. · July 30, 2026 · 8 min read
Scopo's command bar searches your files by meaning, not just by name. Type f: kubernetes cost and a document that never contains those words can still surface, because it's about that. Everything runs on your Mac: Apple's on-device embedding model turns file content into 512-dimensional vectors, and a query becomes a nearest-neighbor search over them. Nothing leaves the machine.
On my Mac that's 391,000 chunk vectors across about 6,500 files. This post is about how we search them on every keystroke, and why the industry-standard answer to that problem turned out to be the wrong one for us.
The sane default
Nearest-neighbor search over hundreds of thousands of vectors is what approximate nearest neighbor (ANN) indexes are for, and HNSW is the default answer. We vendored usearch, a well-regarded single-file HNSW implementation, put SQLite next to it as the durable source of truth, and shipped. Queries came back in under 2 ms. Recall measured 97.8%. Life was good for about a day.
Indexed but unfindable
On adoption day a search for a document I knew existed returned nothing. The file was indexed. Its vector was in the graph, byte-identical to the copy in SQLite. contains(key) said true. A brute-force scan ranked it #2 out of 40,000 for my query, at a strong cosine similarity.
The HNSW search couldn't find it at any fetch depth we tried.
This is a known but under-discussed property of HNSW graphs: a small fraction of nodes end up poorly connected, effectively unreachable from the entry point at practical search widths. On our corpus it measured 1 to 2.5% of nodes, and the membership of that unlucky set is random per build. Rebuild the graph, different files vanish.
What made it fatal for us was a chunking change. Early on, files averaged 172 chunks; losing 2% of chunks was invisible, because the other 170 still matched. Then we moved to token-budget chunking and most files became a single chunk. At one chunk per file, 2% of chunks unreachable means 2% of your files silently don't exist. No error, no crash. Search just quietly lies.
We fixed it the responsible way first: cranked expansion_search to 512 (and learned that usearch silently restores the file's saved value on every view() and load(), so you have to re-apply it after each one), raised expansion_add, and taught the index-maintenance job to self-search every vector and re-insert the unreachable ones. Every rebuild healed the graph. It worked.
But look at what we were now maintaining to keep a graph healthy: an out-of-process fold (the merge step ate so much RAM we re-exec the app binary as a child process just to contain it), delta rebuilds, ghost-key tracking, crash-recovery metadata, and now a verify-and-repair pass that existed because the data structure loses things. All of it machinery in service of one assumption: that the corpus is too big to search exhaustively.
Is it, though?
Measuring brute force
The corpus is a few hundred megabytes of vectors. Modern Apple Silicon moves memory at around 100 GB/s from a few cores. Napkin math says an exhaustive scan should take single-digit milliseconds. So we measured it (Experiment 14 in our lab notes):
| serving strategy | single query | recall@20 vs exact |
|---|---|---|
| HNSW (tuned, healthy) | 1.7 ms | 0.978 |
flat f16 via usearch's own exact_search | 56 ms | 1.0 |
| flat f16, hand-written NEON | 4.9 ms (82 GB/s) | 1.0 |
| flat int8, hand-written NEON | 2.1 ms (97 GB/s) | 0.942 raw |
| flat 4-bit, hand-written NEON | 1.3 ms (77 GB/s) | see below |
Two things jumped out.
First, the library's own exact-search was 10x off the hardware. Its NEON f16 kernel ran at 5.5 GB/s per core, and its int8 path had no NEON kernel at all; it fell back to serial C. I don't say this to dunk on usearch, which is good software. The lesson is narrower: vendored SIMD kernels are optimized for the library author's benchmarks, not your machine. Before concluding brute force is too slow, write the 60-line kernel yourself. Ours is a vdotq_s32 loop that hits memory bandwidth, and it's the least clever code in the app.
Second, at these corpus sizes the quantized scan is faster than the healthy graph, before you count the graph's maintenance machinery at all.
Four bits per dimension
The 4-bit encoding is the part I'd have called reckless a year ago. Each 512-dim vector shrinks from 1 KB of f16 to 256 bytes of codes: unit-normalize, apply a fixed seeded rotation (three rounds of random sign flips + a Walsh–Hadamard transform), then quantize each coordinate against a 16-level Lloyd–Max table.
The rotation is the trick that makes 4 bits survivable, and we took it from the TurboQuant paper (Zandieh, Daliri, Hadian, Mirrokni, 2025). A random rotation smears every vector's energy evenly across dimensions, so each coordinate lands in a near-Gaussian distribution of known scale, which is exactly what a fixed scalar quantizer wants. No codebook training, no data dependence, works online as vectors stream in. We store one extra byte per row (the inverse of the quantized norm) so scores come out cosine-shaped without per-query normalization work.
Raw 4-bit scores drift about 0.002 in cosine distance, enough to shuffle near-ties. We initially fixed that with an exact re-rank of the top candidates from SQLite (recall@20: 0.9988, better than the healthy HNSW). Then we realized we didn't need the re-rank either, for a more interesting reason.
What a graph can never give you
An ANN index answers one question: "what are the K nearest neighbors?" It answers it by not looking at most of the corpus. That's the whole point, and it's also the ceiling.
A full scan already scored everything. So keep all of it. Our kernel now pools scores per file during the scan (mean of each file's best chunks, fused into the same pass), and hands back the complete distribution: every file's pooled score, all ~6,500 of them, in 3 to 4 ms.
Having the whole distribution changed the ranking design more than the index swap did:
- Quantization noise mostly cancels. Pooling a file's best 12 chunks averages the 4-bit jitter down by roughly √12. The re-rank became unnecessary.
- Top-K retrieval is gone as a concept. We used to fetch a fixed top-20 and run heuristics to guess which were real. Now the noise floor is measured per query: take the median and MAD of all pooled scores, accept files whose robust z-score clears a threshold. A vague query has a fat, undifferentiated distribution and honestly returns few semantic results. A sharp query has a clean tail and returns more. Result counts finally tell the truth.
- One scoring domain. Keyword-matched candidates get their semantic score from the same scan table as everything else, so nothing ever compares an f16 cosine against a quantized one.
What shipped
SQLite remains the source of truth. The entire serving layer is now: two memory-mapped flat files of fixed 272-byte rows, one C kernel, one Swift class. The fold, the deltas, the ghosts, the repair pass, and the vendored ANN library are deleted. Twelve hundred lines replaced by about three hundred.
| before (HNSW) | after (flat 4-bit scan) | |
|---|---|---|
| ranking agreement with exact truth (overlap@10) | 0.82 | 0.94 |
| worst-query rank correlation | 0.23 | 0.86 |
| warm query (scan + pool + path lookup) | p50 ~150 ms end-to-end | 3.3–4.8 ms |
| index on disk | 459 MB | 106 MB |
| files that randomly can't be found | ~2% | 0 |
The per-keystroke budget is now dominated by the embedding model itself (~60 ms on the Neural Engine). The search under it is a rounding error, and it's exact: every file is scored on every keystroke, so "indexed but unfindable" is not a reduced probability, it's a deleted category.
The takeaway
If your corpus is millions of vectors on a server, you need an ANN index; this post is not for you. But local-first apps live in a different regime: tens of thousands to low millions of vectors, one query at a time, on hardware with absurd memory bandwidth. In that regime:
- Do the napkin math before adopting a graph. Corpus bytes ÷ bandwidth is your floor, and it's probably milliseconds.
- Don't trust vendored kernels; write the loop. Ours took an afternoon and was 10 to 40x faster than the library's exact path.
- A cheap rotation makes 4 bits respectable. TurboQuant-style rotate-then-quantize needs no training and cut our index by 4x.
- Exhaustive scoring is a feature, not a cost. The full score distribution let us replace retrieval heuristics with per-query statistics.
- If you do run HNSW, census it with self-search.
contains()tells you the vector is stored. It does not tell you search can reach it.
The flat-scan engine ships in the next Scopo release, inside the command bar's file search. Like everything in Scopo's index, the vectors, the scan, and the model never leave your Mac. That constraint is why this design works at all: a corpus small enough to be private is small enough to be searched honestly, all of it, every keystroke.