Question¶

Can hybrid search (combining keyword/structured + semantic vector search) work without a persistent database? Specifically: can frontmatter categories/tags work as structured filters alongside in-memory FAISS vectors? Can FTS run in-memory or with a lightweight on-disk index? What combination gives the best retrieval at 100-1000 doc scale? How does Reciprocal Rank Fusion (RRF) combine results from multiple search methods?

TL;DR¶

Yes. At 100-1000 document scale, the entire hybrid search pipeline (structured frontmatter filtering + full-text search + semantic vector search) can run in-process with no persistent database. The recommended stack is: pre-filter by frontmatter metadata, run SQLite FTS5 :memory: for keyword ranking, run FAISS IndexFlatIP for semantic ranking, fuse with RRF. Total rebuild time on init is under 1 second. Total memory footprint is under 50 MB.

(a) Frontmatter Categories/Tags as Structured Pre-Filters¶

How It Works¶

Picobot's current _search_frontmatter() already parses YAML frontmatter (title, topic, tags) from every KB entry. This is the simplest and most effective structured filter layer: before running any expensive search, narrow the candidate set by topic and tags.

Integration with Vector Search¶

FAISS does not natively support metadata filtering. The standard pattern for small-scale use is pre-filtering: filter documents by metadata first, then search only the matching subset of vectors. At 100-1000 docs, this is trivially fast.

Two practical approaches:

1. ID-based filtering: Maintain a Python dict mapping doc_id -> frontmatter. Filter doc IDs by topic/tag criteria. Then either (a) search a pre-built FAISS index and post-filter results, or (b) build a temporary FAISS index from the filtered subset. At this scale, both are sub-millisecond.

2. Parallel filter + merge: Run frontmatter filtering as its own retrieval signal (which it already is in picobot), then fuse with vector results via RRF. This is architecturally cleaner and avoids coupling the vector index to metadata.

Recommendation for Picobot¶

Keep frontmatter search as a first-class retriever (Phase 1), not just a pre-filter. It captures exact tag matches that semantic search might miss. Feed its ranked results into RRF alongside FTS and vector results. This preserves the existing two-phase architecture while adding semantic search as a third phase.

(b) Full-Text Search Options Without a Dedicated Server¶

Option 1: SQLite FTS5 (Recommended)¶

SQLite's FTS5 extension provides production-quality full-text search with BM25 ranking. It runs entirely in-process with Python's built-in sqlite3 module.

In-memory support: FTS5 virtual tables work in :memory: databases:

import sqlite3

conn = sqlite3.connect(":memory:")
conn.execute("CREATE VIRTUAL TABLE kb USING fts5(doc_id, title, body, tags)")

# Index all KB entries on init
for doc in documents:
    conn.execute("INSERT INTO kb VALUES (?, ?, ?, ?)",
                 (doc.id, doc.title, doc.body, " ".join(doc.tags)))

# Search with BM25 ranking
results = conn.execute("""
    SELECT doc_id, title, bm25(kb) as score
    FROM kb WHERE kb MATCH ?
    ORDER BY bm25(kb)
    LIMIT 10
""", (query,)).fetchall()

Key properties: - BM25 ranking built in (note: lower scores = better relevance; use ORDER BY bm25(kb) ascending, or negate for descending sort) - Column weights supported: bm25(kb, 0.0, 2.0, 1.0, 1.5) to weight title higher than body - Porter stemming available: tokenize="porter unicode61" - Rebuilds in-memory from 1000 markdown files in ~50-100ms - Zero external dependencies (ships with Python's sqlite3) - Caveat: Some Python distributions may not include FTS5. Check with conn.execute("CREATE VIRTUAL TABLE test USING fts5(x)") at init and fall back if unavailable

BM25 scoring detail: FTS5's bm25() returns negative scores where more negative = more relevant. This is an implementation quirk. To get conventional positive-higher-is-better scores, negate the result: score = -bm25_score.

Option 2: Whoosh (Pure Python, In-Memory via RamStorage)¶

Whoosh is a pure-Python full-text search library with Lucene-inspired architecture. It supports fully in-memory indexing via RamStorage:

from whoosh.fields import Schema, TEXT, ID
from whoosh.filedb.filestore import RamStorage
from whoosh.qparser import QueryParser

schema = Schema(doc_id=ID(stored=True), title=TEXT(stored=True), body=TEXT)
storage = RamStorage()
index = storage.create_index(schema)

writer = index.writer()
for doc in documents:
    writer.add_document(doc_id=doc.id, title=doc.title, body=doc.body)
writer.commit()

with index.searcher() as searcher:
    query = QueryParser("body", index.schema).parse(user_query)
    results = searcher.search(query, limit=10)

Key properties: - Pure Python, no C extensions needed - BM25F scoring by default - Rich query syntax (AND, OR, wildcards, phrase, fuzzy) - In-memory via RamStorage, or file-based for persistence - Drawback: unmaintained upstream (last release 2015); community fork whoosh-reloaded exists but activity is low - Drawback: slower than FTS5 for indexing and search due to pure-Python implementation

Option 3: tantivy-py (Rust-Backed, High Performance)¶

tantivy-py provides Python bindings for the Tantivy search engine (Rust). It supports in-memory indexes:

import tantivy

schema_builder = tantivy.SchemaBuilder()
schema_builder.add_text_field("title", stored=True)
schema_builder.add_text_field("body", stored=True)
schema = schema_builder.build()

index = tantivy.Index(schema)  # in-memory by default
writer = index.writer()
writer.add_document(tantivy.Document(title="Example", body="Content here"))
writer.commit()

searcher = index.searcher()
query = index.parse_query("content", ["title", "body"])
results = searcher.search(query, limit=10)

Key properties: - Rust performance (significantly faster than Whoosh) - BM25 scoring - In-memory by default; pass path= for on-disk persistence - Requires pre-built binary wheels (available for macOS, Linux, Windows on Python 3.9+) - Adds ~15 MB binary dependency - More actively maintained than Whoosh

Option 4: rank-bm25 / bm25s (Pure Python BM25, No Index)¶

For the simplest possible keyword search, rank-bm25 or bm25s implement BM25 scoring directly over tokenized documents in memory, with no index structure at all:

from rank_bm25 import BM25Okapi

corpus = [doc.body.split() for doc in documents]
bm25 = BM25Okapi(corpus)
scores = bm25.get_scores(query.split())
top_n = bm25.get_top_n(query.split(), documents, n=10)

Key properties: - Minimal dependencies (just numpy for bm25s) - No index to build or maintain - bm25s claims up to 500x speedup over rank-bm25 - No stemming, phrase queries, or field weighting unless you implement them yourself - Good enough for 100-1000 docs where index overhead is not justified

Comparison Table¶

Recommendation¶

SQLite FTS5 is the clear winner for picobot: - Zero new dependencies (Python's sqlite3 includes it) - Production-quality BM25 with column weighting - Porter stemming for morphological matching - Trivial to rebuild in-memory on each picobot init - Battle-tested at any scale

If FTS5 is unavailable in the Python build, fall back to bm25s (one lightweight dependency, good performance).

(c) Optimal Combination at 100-1000 Document Scale¶

Recommended Architecture: Three Retrievers + RRF¶

User Query
    |
    v
+---+---+---+
|   |   |   |
v   v   v   v

Phase 1:          Phase 2:           Phase 3:
Frontmatter       SQLite FTS5        FAISS IndexFlatIP
(tags/title       (BM25 on body)     (cosine similarity
 matching)                            on embeddings)
    |                 |                    |
    v                 v                    v
 ranked list       ranked list         ranked list
    |                 |                    |
    +--------+--------+-------------------+
             |
             v
     Reciprocal Rank Fusion (RRF)
             |
             v
      Final ranked results

Resource Budget at 1000 Documents¶

Notes on the budget: - The embedding model dominates memory. With a local model like all-MiniLM-L6-v2 (384 dimensions, ~22M parameters), expect ~80 MB for the model itself. If using API-based embeddings (OpenAI, Anthropic), this drops to zero. - FAISS vector storage for 1000 docs at 384 dimensions = 1000 x 384 x 4 bytes = ~1.5 MB. Negligible. - Cold start is dominated by embedding model load (~2s). Index rebuilds are ~200ms total for all three indexes. If the model is already loaded (warm start), total init is under 500ms. - Query-time embedding of the query takes 5-20ms on CPU. The search itself (all three retrievers + RRF) is under 10ms.

FAISS Index Choice¶

At 100-1000 documents, use IndexFlatIP (flat inner product / cosine similarity on normalized vectors). There is no reason to use approximate indexes (IVF, HNSW) at this scale:

• Flat index requires no training
• Exhaustive search over 1000 384-dim vectors takes <1ms
• No tuning parameters
• Supports add() and remove_ids() for incremental updates
• Use IndexIDMap(IndexFlatIP(dim)) to maintain doc_id mappings

import faiss
import numpy as np

dim = 384
index = faiss.IndexIDMap(faiss.IndexFlatIP(dim))

# Add documents (vectors must be L2-normalized for cosine similarity)
vectors = np.array([doc.embedding for doc in documents], dtype=np.float32)
faiss.normalize_L2(vectors)
ids = np.array([doc.id_int for doc in documents], dtype=np.int64)
index.add_with_ids(vectors, ids)

# Search
query_vec = embed(query).reshape(1, -1).astype(np.float32)
faiss.normalize_L2(query_vec)
scores, doc_ids = index.search(query_vec, k=10)

Init Strategy¶

Rebuild all three indexes on picobot init (or on first search after startup):

1. Walk KB directory, parse all markdown files, extract frontmatter + body
2. Build frontmatter lookup dict (already done in current code)
3. Insert all docs into SQLite FTS5 :memory: table
4. Compute embeddings for all docs (or load cached embeddings from .picobot/cache/embeddings.npz)
5. Build FAISS index from embedding vectors

Embedding cache is important: re-embedding 1000 documents on every init wastes time. Cache embeddings keyed by content hash. Only re-embed changed documents.

~/.picobot/cache/embeddings.npz   # numpy archive: {content_hash -> vector}

(d) Reciprocal Rank Fusion (RRF)¶

How RRF Works¶

RRF is a rank aggregation method that combines ranked lists from multiple retrievers into a single ranking. For each document d that appears in any retriever's results:

RRF(d) = SUM over all retrievers r of: 1 / (k + rank_r(d))

Where: - rank_r(d) is the 1-based rank of document d in retriever r's results - k is a smoothing constant (typically 60) - If document d does not appear in retriever r's results, that term is 0

Why k=60?¶

The original RRF paper tested values from 1 to 100. Values below 10 let a single top-ranked result dominate (rank 1 gets vastly more weight than rank 2). Values above 100 flatten the curve too much (rank 1 and rank 20 are nearly equal). The value 60 was empirically found to balance these extremes across diverse datasets. At k=60, rank 1 gets a score of 1/61 = 0.0164, rank 2 gets 1/62 = 0.0161 -- a gradual decline that rewards consensus across retrievers.

Implementation for Picobot¶

RRF is trivial to implement in-process. Here is a complete implementation:

def rrf_fuse(
    ranked_lists: list[list[str]],
    k: int = 60,
    top_n: int = 10,
) -> list[tuple[str, float]]:
    """
    Reciprocal Rank Fusion over multiple ranked lists of doc IDs.

    Args:
        ranked_lists: List of ranked lists, each containing doc_id strings
                      ordered from most to least relevant.
        k: Smoothing constant (default 60).
        top_n: Number of results to return.

    Returns:
        List of (doc_id, rrf_score) tuples, sorted by score descending.
    """
    scores: dict[str, float] = {}
    for ranked_list in ranked_lists:
        for rank_pos, doc_id in enumerate(ranked_list, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank_pos)

    sorted_results = sorted(scores.items(), key=lambda x: x[1], reverse=True)
    return sorted_results[:top_n]

Example¶

Suppose three retrievers return these rankings for a query:

Frontmatter: [doc_A, doc_B, doc_C]
FTS5 BM25:   [doc_B, doc_D, doc_A]
FAISS:       [doc_D, doc_A, doc_B]

RRF scores (k=60): - doc_A: 1/61 + 1/63 + 1/62 = 0.0164 + 0.0159 + 0.0161 = 0.0484 - doc_B: 1/62 + 1/61 + 1/63 = 0.0161 + 0.0164 + 0.0159 = 0.0484 - doc_C: 1/63 + 0 + 0 = 0.0159 - doc_D: 0 + 1/62 + 1/61 = 0.0161 + 0.0164 = 0.0325

Final ranking: [doc_A, doc_B, doc_D, doc_C] (A and B tie; D beats C because it appeared in two lists)

Why RRF Over Weighted Score Combination?¶

1. No score normalization needed: BM25 scores, cosine similarities, and frontmatter overlap ratios are on completely different scales. Combining raw scores requires per-retriever normalization (min-max, z-score), which is fragile. RRF only needs rank order.

2. Robust to retriever quality differences: If one retriever is noisy, its bad results get low RRF contributions. If a doc is genuinely relevant, multiple retrievers will rank it highly, amplifying its RRF score.

3. Zero tuning required: k=60 works well out of the box. No need to tune per-retriever weights (though you could extend RRF with per-retriever weight multipliers if needed).

4. Sub-millisecond computation: For 3 retrievers returning 10 results each, RRF involves at most 30 score additions. Negligible overhead.

Practical Considerations¶

• Retriever result count: Each retriever should return enough candidates (e.g., top 20) to give RRF good coverage. If a retriever returns only 3 results, it limits fusion quality.
• Degenerate case: If only one retriever returns results for a query (e.g., the query has no good keyword matches), RRF gracefully degrades to that retriever's ranking.
• Optional: weighted RRF: Multiply each retriever's contribution by a weight: w_r / (k + rank). This lets you bias toward semantic or keyword results. Start with equal weights; tune only if needed.

Summary: Recommended Architecture for Picobot¶

Stack¶

No Persistent Database Required¶

All three indexes rebuild from the KB markdown files on init: - Frontmatter: parse YAML headers (already done) - FTS5: insert text into :memory: SQLite table - FAISS: load cached embeddings or compute fresh ones

Total cold-start time with cached embeddings: ~300-500ms for 1000 docs. Total cold-start time without cache (first run): ~30-60s for 1000 docs (dominated by embedding computation).

Migration Path from Current Code¶

The current KnowledgeBase.search() method runs Phase 1 (frontmatter) then Phase 2 (substring fulltext) sequentially, merging without scores. The upgrade path:

1. Replace _search_fulltext() substring matching with SQLite FTS5 BM25 ranking
2. Add Phase 3: FAISS semantic search
3. Run all three phases, collect ranked lists
4. Fuse with RRF instead of the current "append if not seen" merge logic
5. Cache embeddings to disk, rebuild FTS5 index on init

This is a non-breaking change to the search() public API -- the return format stays the same, only result quality improves.