Question: Can picobot use a no-DB in-memory approach — load pre-computed embeddings on init, build a FAISS flat index in memory, and skip any persistent database entirely?

Verdict: Yes, absolutely. At picobot's scale (100-1000 docs), a flat in-memory index is the correct choice. The entire approach fits comfortably in memory, queries complete in sub-millisecond time, and the flat brute-force index requires zero training or tuning. FAISS works but is arguably over-engineered for this scale — numpy dot product or a lighter library like usearch would also suffice.

(a) faiss-cpu: Install Size, Dependencies, Heavy Libs¶

Package: faiss-cpu v1.13.2 (Dec 2025, MIT + BSD-3-Clause)

Wheel sizes (download):

Dependencies: numpy, packaging. That's it. No torch, no BLAS install required — on macOS it uses the Accelerate framework (ships with the OS), on Linux it bundles OpenBLAS in the wheel. No heavy transitive dependencies.

Installed size on disk: Roughly 2-3x the wheel size once unpacked (the wheel is compressed). Expect ~10-70 MB depending on platform. The Linux wheel is the largest because it bundles OpenBLAS.

Assessment: Lightweight enough. The macOS ARM64 wheel is only 3.5 MB — smaller than many utility libraries. It does not pull in torch, CUDA, or any GPU libraries. The packaging dependency is already present in virtually every Python environment.

(b) IndexFlatIP vs IndexFlatL2 for Cosine Similarity¶

Recommendation: Use IndexFlatIP with pre-normalized vectors.

The relationship between cosine similarity and inner product is:

cosine_similarity(a, b) = dot(a, b) / (||a|| * ||b||)

If both vectors are L2-normalized (unit length), then ||a|| = ||b|| = 1, and:

cosine_similarity(a, b) = dot(a, b) = inner_product(a, b)

Therefore: - IndexFlatIP (inner product) computes cosine similarity directly on normalized vectors - IndexFlatL2 (Euclidean distance) does NOT give cosine similarity — it gives L2 distance, which has a monotonic relationship to cosine on normalized vectors but returns different numeric values

Implementation pattern:

import faiss
import numpy as np

# Normalize vectors to unit length
faiss.normalize_L2(vectors)  # in-place normalization

# Build index
index = faiss.IndexFlatIP(768)  # 768 dimensions
index.add(vectors)

# Query (also normalize query)
faiss.normalize_L2(query)
distances, indices = index.search(query, k=10)
# distances are cosine similarities (higher = more similar)

Note: With IndexFlatIP on normalized vectors, returned distances ARE cosine similarity values in [0, 1] for positive embeddings (or [-1, 1] in general). Higher is better. With IndexFlatL2, lower is better and the values are Euclidean distances.

(c) Query Latency at Various Scales (768-dim vectors)¶

For a flat (brute-force) index, query time is O(N * d) where N = number of vectors and d = dimensions. The operation is a matrix-vector multiply.

Estimated single-query latency for IndexFlatIP, 768 dimensions:

These estimates are based on FAISS's SIMD-optimized inner product computation. FAISS uses AVX2/AVX-512 on x86 and NEON on ARM, which process multiple float32 operations per cycle. At 1,000 docs, the entire computation is ~3 MB of sequential memory reads and ~768K multiply-add operations — trivially fast for modern CPUs.

For picobot's 100-1000 doc range, query latency is effectively zero (sub-millisecond). The LLM API call that follows will take 500-5000ms, making vector search latency completely irrelevant.

(d) Memory Footprint¶

Each vector: 768 dimensions * 4 bytes (float32) = 3,072 bytes = 3 KB

FAISS flat index overhead is minimal — it stores vectors in a contiguous array with only a small amount of metadata. The overhead is roughly 10-15% beyond the raw vector data.

For picobot at 1,000 docs: 3 MB. This is negligible. The Python process itself uses 20-50 MB. The embedding model (if loaded locally) would use 500 MB-2 GB. The vector index is a rounding error.

(e) Init Time to Build Index from Pre-loaded Vectors¶

IndexFlatIP and IndexFlatL2 require zero training. Building the index is just a memcpy of the vectors into a contiguous array.

Estimated init times:

The operation is:

vectors = np.load("embeddings.npy")  # load from disk
faiss.normalize_L2(vectors)           # normalize in-place
index = faiss.IndexFlatIP(768)
index.add(vectors)                    # just a memcpy

Loading embeddings from disk (reading the .npy file) will dominate init time. For 1,000 docs at 3 MB, even a slow disk reads that in <10 ms. Total init for picobot's scale: under 20 ms.

(f) When Does Flat Brute-Force Break Down?¶

FAISS official guidelines (from the wiki):

• < 10K vectors: Use Flat. It is fast enough, provides exact results, and needs no tuning.
• 10K - 100K vectors: Flat still works but consider IVF for faster queries if latency matters.
• 100K - 1M vectors: IVF recommended. Use IVF{K},Flat where K = 4sqrt(N) to 16sqrt(N).
• 1M+ vectors: IVF with HNSW coarse quantizer. Use IVF65536_HNSW32,Flat etc.

Practical breakpoints:

For picobot (100-1000 docs): Flat is not just acceptable, it is the correct choice. Any approximate index (IVF, HNSW) would add complexity, tuning parameters, and training time for zero benefit at this scale.

(g) FAISS Index Serialization to Disk¶

Yes, FAISS indexes can be serialized for fast reload:

import faiss

# Save to disk
faiss.write_index(index, "kb_index.faiss")

# Load from disk (fast — memory-mapped read)
index = faiss.read_index("kb_index.faiss")

# Alternative: serialize to numpy array (useful for embedding in other formats)
serialized = faiss.serialize_index(index)
np.save("kb_index.npy", serialized)

# Deserialize
index = faiss.deserialize_index(np.load("kb_index.npy"))

Read-only memory-mapped loading is also supported for very large indexes:

index = faiss.read_index("kb_index.faiss", faiss.IO_FLAG_MMAP | faiss.IO_FLAG_READ_ONLY)

File sizes for a flat index are essentially the raw vector data plus a small header: - 1,000 docs * 768 dims * 4 bytes = ~3 MB on disk - Load time from SSD: < 5 ms

For picobot: Serialization is useful for avoiding re-computation of embeddings on restart. However, at 1,000 docs, rebuilding the index from a numpy file is so fast (< 20 ms) that serializing the FAISS index specifically provides minimal advantage over just storing the raw embeddings in a .npy sidecar file.

(h) Alternatives to FAISS for Simple In-Memory Vector Search¶

1. Pure NumPy (no additional dependency)¶

import numpy as np

# Pre-normalize embeddings
embeddings = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)

# Query: just matrix multiply
query_norm = query / np.linalg.norm(query)
similarities = embeddings @ query_norm  # cosine similarities
top_k = np.argsort(similarities)[-k:][::-1]

Pros: - Zero additional dependencies (numpy is already required by everything) - Completely transparent — 3 lines of code - At 1,000 docs, performance is ~100-500 us (fast enough) - Easy to debug, test, and understand

Cons: - No SIMD optimization beyond what numpy/BLAS provides (still good, but FAISS is faster) - No built-in serialization (but numpy save/load is trivial) - At 10K+ docs, noticeably slower than FAISS

Verdict for picobot: This is the strongest contender. At 100-1000 docs, numpy is fast enough and adds zero dependencies.

2. USearch (unum-cloud/usearch)¶

• Install: < 1 MB wheel (macOS ~400 KB, Linux ~2 MB), no required dependencies
• Algorithm: HNSW (approximate, but exact at small scale with high ef_search)
• Metrics: L2, cosine, inner product, plus user-defined
• Claimed: 10-100x faster than FAISS in benchmarks (though mainly at large scale)
• API: Simple — Index(ndim=768), .add(), .search()
• Persistence: Built-in save/load

Verdict: Excellent lightweight option if you want a proper vector search library without FAISS's weight. Overkill for 1,000 docs but future-proofs nicely.

3. HNSWlib (nmslib/hnswlib)¶

• Install: Source distribution only (36 KB), compiles from C++ on install. Header-only with no dependencies beyond C++11.
• Algorithm: HNSW (approximate nearest neighbor)
• Metrics: L2, inner product, cosine
• Performance: Very fast for ANN search, well-tested
• API: Straightforward Python bindings
• Note: Last release v0.8.0 was Dec 2023 — somewhat stale

Verdict: Good library but requires C++ compilation on install (can fail on some systems). Stale maintenance is a concern. USearch or Voyager are better modern alternatives.

4. Voyager (spotify/voyager)¶

• Install: 200 KB - 4.5 MB wheel depending on platform. Pre-built wheels for all major platforms.
• Algorithm: HNSW (built on hnswlib internals)
• Dependencies: NumPy only
• Production: Used at Spotify for hundreds of millions of queries/day
• Metrics: Cosine, Euclidean, inner product
• API: Clean Python API with save/load support

Verdict: Well-maintained, production-proven, lightweight. A solid choice if you want something between numpy and FAISS. But again, HNSW is unnecessary complexity for 1,000 docs.

Comparison Matrix¶

Recommendation for Picobot¶

Tier 1 — Start here: Pure numpy dot product. - At 100-1000 docs, embeddings @ query completes in < 500 us - Zero additional dependencies - Completely debuggable and testable - Store embeddings as a numpy .npy sidecar file next to the KB - Total code: ~20 lines for the entire search module

Tier 2 — If you want a real library: faiss-cpu with IndexFlatIP. - Marginally faster than numpy (SIMD-optimized inner product) - Well-known, extensively documented, battle-tested - Flat index needs zero tuning — just add() and search() - Adds ~3.5 MB dependency (macOS ARM64) - Worth it if you anticipate growing past 5K docs

Tier 3 — If you want minimal footprint + future scale: usearch. - Smallest install (400 KB) - No dependencies at all - HNSW algorithm scales to millions of vectors if needed - Less widely known than FAISS, but actively maintained

Architecture sketch (numpy approach):

KB init:
  1. Load markdown files, parse frontmatter
  2. Load embeddings from sidecar file (embeddings.npy)
     - If sidecar missing or stale, compute embeddings and save
  3. Normalize embeddings in memory
  4. Ready for search

KB search:
  1. Embed query string (API call to embedding model)
  2. Normalize query embedding
  3. scores = embeddings @ query  (numpy dot product)
  4. Return top-k doc indices sorted by score

No database. No server. No background process. ~3 MB memory.