How we scaled Miriel's confidential video search from one machine to a 20-node fleet, ingested 100 billion vectors at 37.5 million/sec, and kept search latency flat the whole way up, with every vector encrypted the entire time.
TL;DR
- 100 billion vectors indexed end-to-end on a 20-machine fleet. Every vector encrypted at rest, in transit, and in use (CyborgDB DiskIVF).
- 37.5M vectors/sec aggregate ingest (1.88M/sec/box), near-perfectly linear in fleet size.
- 62.4 TB of encrypted vector index plus 113 billion rows of Postgres metadata. 0 failed shards across 2,000 shards.
- Search latency is scale-invariant: per-shard warm latency held at ~199 ms p50 / 258 ms p99 from 5B (1 box) to 100B (20 boxes), measured over 2 million queries.
- Recall, on real embeddings: 0.93 recall@10 vs exact KNN at our operating point (5.5B-vector dataset); the exact cross-shard merge holds it fleet-wide.
- Total run: ~11.3 hours end-to-end. 9.8 h to build the index, plus a one-time 1.5 h cold-start warm-up. (The 2M-query benchmark itself ran in a couple of minutes.)
When we set out to test whether our video-search stack could hold up at extreme scale, the real question wasn't "can we store 100B vectors." It was "does anything break or slow down when we get there." Nothing did.
Why this is hard#
Miriel turns video into searchable meaning. Every sampled frame becomes a series of 256-dimensional vectors: a whole-frame embedding plus one for each salient region, crop, and patch it contains, so a busier frame yields more vectors. "Find every frame that looks like this" becomes a nearest-neighbor search over billions of them. Two properties make it genuinely hard:
-
Confidentiality. The embeddings are the customer's footage in latent form. Leaking them leaks the video. So the index has to be encrypted at rest and searched without ever materializing plaintext vectors. We use CyborgDB, a confidential vector database whose encrypted IVF index keeps vectors ciphertext on disk and in memory.
-
Scale. A serious deployment is tens of billions of frames. That's two scaling axes at once: make a single machine as dense and fast as possible (vertical), then make many machines add up without coordination tax (horizontal). Both have to hold while keeping search fast.
This post walks the system top to bottom, then shows what we measured at 100B.
The system, end to end#
Coordinator. The brain. It holds the run-list, slices ingestion into work items onto an SQS queue, and autoscales the ingest fleet to drain it. At query time it fans a search out to the storage fleet and merges the results.
Ingestor fleet. An autoscaling group of workers. Each pulls a video off the queue, decodes it, samples frames, and runs the embedding model. Each frame yields a series of 256-dim vectors (whole frame, regions, crops, patches), how many depending on the frame's content. It upserts them (already encrypted) straight into the storage shards, writing frame metadata (video, camera, S3 URI, score, timestamp) to Postgres alongside. The work is independent per frame — no worker ever coordinates with another — so throughput scales by just adding workers (a textbook embarrassingly parallel workload).
Storage fleet. This is where the scaling work lives. Each machine runs a set of CyborgDB DiskIVF shards (50M encrypted vectors each, fp16, disk-resident), plus a partitioned Postgres table holding the frame metadata for those shards. A search fans out to every box; each box searches its local shards and returns its top candidates with exact distances; the coordinator does an exact cross-box distance-merge to a global top-K and hydrates the metadata.
The key architectural decision: the storage fleet is shared-nothing. No box talks to another during ingest or search. That's what makes horizontal scaling clean, and it's why, as you'll see, throughput is linear and latency is flat.
Vertical scaling: making one machine dense and fast#
Before adding machines, we squeezed a single box. The shard config that won:
DiskIVF, storage_precision=fp16, n_lists=7071 (≈√50M), n_probes=8,
rerank_mult=20. fp16 halves the index footprint versus fp32 with no
measurable recall cost at our dimensionality; DiskIVF keeps the index
disk-resident so a 50M-vector shard doesn't have to fit in RAM.
The non-obvious bottleneck was disk bandwidth. Both ingest and training stream the full shard off disk, so a single EBS volume starves them. Striping five volumes into one RAID0 array roughly doubled effective bandwidth, and ingest throughput with it:
That single change (1 volume → 5×RAID0) took per-box ingest from ~0.96M to ~1.88M vectors/sec, and it protects the (disk-heavy) training phase too. Each box now holds 5 billion encrypted vectors and ingests at nearly 2M/sec.
Horizontal scaling: making 20 machines add up#
With a strong single box, scaling out should be "multiply by N" if the architecture is genuinely shared-nothing. It is. We ran 20 × c7i.16xlarge, each holding 5B vectors (100 shards × 50M), for 100B total, and measured the aggregate ingest as we accumulated boxes:
37.5M vectors/sec aggregate, sitting right on the ideal-linear line. And it's not a few fast boxes carrying the average. Every machine pulls its weight:
Per-box ingest landed in a tight 1.77M–2.03M/sec band (mean 1.88M). The full build:
| Phase | Result |
|---|---|
| Ingest | 37.5M vec/sec aggregate, 100B vectors |
| Encrypted vector index | 62.4 TB on disk (fp16 DiskIVF) |
| Metadata | 113B Postgres rows |
| Train | ~7.7 h/box (2,000 shards, 8-way concurrent), 0 failures |
| Wall-clock | 9.8 h build + 1.5 h one-time warm-up = ~11.3 h end-to-end (2M-query benchmark ran in minutes) |
2,000 shards trained across 20 machines with zero failed shards. That's the entire build (ingest, encrypt, train, and index 100 billion vectors) in roughly a working day, on commodity CPU instances.
Search at 100B: latency that doesn't care how big you are#
Throughput is the easy half. The real question is whether search degrades as the dataset grows. To answer it rigorously we ran a 2-million-query benchmark: 1,000 warm queries against each of the 2,000 shards.
Per-shard warm latency: 199 ms median, 239 ms p95, 258 ms p99, tightly distributed across all 2,000 shards. Because the storage fleet is shared-nothing and the cross-box merge is exact, a shard answers in the same ~199 ms whether it lives in a 5B deployment or a 100B one. Comparing the single-box 5B run to the full 100B fleet:
A 20× increase in dataset size moved median latency by ~21 ms. Search cost is a property of a shard, not of the dataset. You scale the dataset by adding shards/boxes in parallel; the coordinator fans out and merges, so end-to-end latency tracks the slowest shard in the fan-out plus a cheap merge, not the total vector count.
A note on the measurement: the 199 ms / 258 ms figures are per-shard warm latency, measured 2M times. End-to-end latency for a query depends on how many shards it fans out to (the coordinator merges them in parallel). We report the per-shard number because it's the scale-invariant unit and what we measured directly. No extrapolation.
Confidential by default: encrypted at rest and in use#
Every one of those 100 billion vectors is encrypted, not just at rest but also in use. The embeddings are the customer's footage in latent form, so protecting them only on disk isn't enough: they have to stay protected while we search.
CyborgDB enforces this in three places. At rest, the vectors, IDs, metadata, and index nodes are stored as ciphertext (AES-256-GCM with per-record IVs and AEAD authentication), so disk access reveals nothing. In transit, payloads are encrypted client-side before they ever hit the network. In use, search runs directly over the encrypted index: the index itself is never decrypted, and only the final answer to a query is briefly decrypted. Keys stay in the customer's custody (BYOK/HYOK). Nothing on the server side can decrypt without them.
Maintaining encryption in use, while searching, is the unusual part. A typical vector database has to hold plaintext vectors in memory to search them. CyborgDB does not, so at no point in this 100-billion-vector run did a plaintext embedding exist, in storage or during search. The cost is small where it counts: CyborgDB's published encryption overhead is about 1% on index build and under 15% on search. The build is what dominates a run like this, so the 9.8 h above is essentially its unencrypted cost. The search overhead wasn't an issue either: even with encryption on the whole time, warm search held around 199 ms.
Validating real-world recall#
The 100B run above used random/synthetic vectors. That's the right tool for stressing throughput, scale, and latency, but it says nothing about recall (random vectors have no cluster structure for a nearest-neighbor search to find). Recall has to be measured on real embeddings, so we ran a separate experiment.
One property makes that tractable: with per-shard ANN search plus an exact global merge, horizontal scale does not add another recall-losing stage. Each shard runs its local IVF search; the coordinator then merges returned candidates by true distance. There is no approximate global routing layer where more shards can dilute the result.
That is why the 5.5B real-video recall test matters for the 100B deployment: horizontal scale adds more independent shards while preserving the same merge contract. We validate quality on real embeddings, then validate throughput, training, and latency at 100B.
The experiment. We ingested 5.5B real video embeddings (Miriel's own
image-family vectors: MobileCLIP2 + Nomic-vision, MRL-truncated to dim-256)
across production-shaped 50M-vector DiskIVF shards (n_lists ≈ √50M, fp16,
encrypted), and compared results against exact KNN ground truth over 500
sampled queries, sweeping n_probes × rerank_mult to map the operating
curve:
| n_probes | rerank_mult | recall@10 | recall@100 |
|---|---|---|---|
| 8 | 20 | 0.708 | 0.672 |
| 8 | 50 | 0.925 | 0.904 |
| 16 | 20 | 0.734 | 0.698 |
| 16 | 50 | 0.926 | 0.906 |
| 32 | 20 | 0.735 | 0.699 |
| 32 | 50 | 0.927 | 0.907 |
At the higher-recall operating point (n_probes=16, rerank_mult=50):
recall@10 = 0.93, recall@100 = 0.91 against exact KNN, on an encrypted,
disk-resident index.
The sweep makes the dominant lever obvious: it's rerank_mult, not
n_probes. Lifting rerank 20→50 takes recall@10 from ~0.73 to ~0.93,
while tripling n_probes (8→32) barely moves it. Rerank re-scores a few
more candidates with exact distances, cheap relative to the IVF probe, so
it's the knob that buys recall. (The throughput benchmark earlier ran the
lighter rerank_mult=20, so the ~199 ms latency figure lives at that point
while 0.93 recall lives at 50: two points on the same quality/latency curve,
not the same measurement.)
These are complementary proofs: 5.5B real video embeddings prove recall on the data distribution that matters; the 100B run proves the shared-nothing system keeps ingest, training, and latency stable at full scale. Because the global merge is exact, horizontal scale does not introduce a second recall penalty.
Summary#
| Vectors indexed | 100 billion (encrypted) |
| Fleet | 20 × c7i.16xlarge, shared-nothing |
| Ingest | 37.5M vec/sec (1.88M/box), linear scaling |
| Vector index | 62.4 TB (fp16 DiskIVF) + 113B Postgres metadata rows |
| Train | 2,000 shards, ~7.7 h/box, 0 failures |
| Search (per-shard, warm) | 199 ms p50 / 258 ms p99, scale-invariant |
| Recall (real embeddings) | 0.93 recall@10 / 0.91 recall@100 vs exact KNN (5.5B dataset, n_probes=16/rerank=50) |
| Queries benchmarked | 2 million |
| Confidentiality | encrypted at rest, in transit, and in use; searched on ciphertext |
| Run time | ~11.3 h (9.8 h build + 1.5 h one-time cold-start warm-up; 2M queries ran in minutes) |
We stored 100 billion vectors and showed a shared-nothing, encrypted vector-search system can ingest, train, and query at that scale without throughput collapse, shard failures, or per-shard latency growth: a confidential video-search index at 100-billion-vector scale, built end to end in roughly a working day on commodity hardware, encrypted at rest, in transit, and in use, with search latency staying flat as the dataset grows.
