Research Date: 2026-02-25 Lines of Enquiry: 7/7 completed Implementation Plan: implementation-plan.md

Executive Summary¶

Picobot can have high-quality semantic search with zero database dependency and under 50 MB memory overhead. The recommended architecture:

1. Nomic embeddings (nomic-embed-text-v1.5) — MTEB 62.28, Apache 2.0, local+remote from a single API call, 768d (full quality — the memory savings of 256d Matryoshka are negligible at picobot's scale)
2. HyPE indexing — generate 5-10 hypothetical questions per KB entry at write time, embed those (not the document). Up to +42pp context precision for intent-based agent queries
3. No database — cache embeddings in a gitignored .npz file, load into numpy on init, brute-force cosine similarity. 1000 docs at 768d = 3 MB memory, <0.1ms query time
4. Three-retriever RRF fusion — frontmatter tags + SQLite FTS5 in-memory + numpy vector search, combined via Reciprocal Rank Fusion (~15 lines of Python)
5. Don't store embeddings in frontmatter — re-embedding 1000 docs takes 2-6 seconds on CPU; frontmatter storage adds 273% file size overhead and destroys git diffs for a 3-second optimization

Embeddings are always on — nomic[local] is a required dependency (~5 MB, plus ~140 MB GGUF model downloaded on first use). Graduate to faiss-cpu at 5K+ docs if needed.

1. Embedding Model: Nomic Wins¶

Why Nomic¶

• +6 MTEB points over MiniLM at full dimensionality — meaningful retrieval quality gap
• 8,192 token context vs MiniLM's 256 — can embed entire KB documents without truncation
• Local and remote from a single API: inference_mode='local' uses GPT4All (no API key), inference_mode='remote' uses Nomic Atlas (faster for batch), inference_mode='dynamic' auto-selects. Embeddings are compatible across modes
• Apache 2.0 with open training data — fully open, no usage restrictions
• Only viable model that works both offline and via API with the same vectors

Why 768 dimensions (not 256)¶

At picobot's scale (100-1000 docs, up to ~6000 vectors with HyPE), the cost of 768d over 256d is negligible:

768d buys +1.24 MTEB points for 12 MB of RAM. No reason to sacrifice quality at this scale.

Recommended configuration¶

# Local mode (default, no API key needed)
from nomic import embed
output = embed.text(texts, model='nomic-embed-text-v1.5', inference_mode='local', dimensionality=768)

# Remote mode (faster for batch re-embedding)
output = embed.text(texts, model='nomic-embed-text-v1.5', inference_mode='remote', dimensionality=768)

2. Don't Store Embeddings in Frontmatter¶

The Math¶

Why Not¶

1. Git diffs become unreadable — every content edit produces a multi-KB base64 diff that drowns actual changes
2. Repository bloat — 500 docs edited 10 times each = ~20 MB of embedding delta in git history (base64 has poor delta compression)
3. The model must load anyway — you need it at query time for embedding the search query, so frontmatter only defers the load cost, doesn't eliminate it
4. 2-6 seconds is not a problem — this happens once on init, not per-query

What to Do Instead¶

Cache embeddings in a gitignored sidecar file:

# Save: ~/.picobot/data/embeddings.npz (gitignored)
np.savez_compressed('embeddings.npz',
    vectors=embeddings,        # (N, 768) float32
    doc_ids=doc_ids,           # string array
    content_hashes=hashes)     # for invalidation

# Load: instant on subsequent inits
data = np.load('embeddings.npz')
# Only re-embed docs where content_hash changed

Content-hash based invalidation: hash each doc's body, compare to stored hashes, only re-embed changed docs. After the first init, subsequent startups re-embed only new/modified entries.

3. What to Embed: HyPE (Hypothetical Questions)¶

The Problem¶

Agent queries are intent-based: "find information about error recovery patterns", "what do I know about deployment". These match poorly against document text because the linguistic form of a question is very different from the form of an answer.

The Solution: Hypothetical Question Embedding (HyPE)¶

For each KB entry, generate 5-10 questions the document could answer. Embed those questions. At query time, match query against question embeddings (question-to-question matching is much higher fidelity than question-to-document).

Benchmarks: HyPE shows up to +42 percentage points in context precision and +45 percentage points in claim recall vs standard document embedding.

Implementation¶

At kb store time, after writing the markdown file:

QUESTION_PROMPT = """Given this knowledge base entry, generate 5-10 questions
that an AI agent might ask when it needs this information. Focus on:
- Task-oriented questions ("How do I...", "What's the best way to...")
- Lookup questions ("What is...", "Where is...")
- Decision questions ("Should I...", "When should I...")

Document:
{title}
{body}

Questions (one per line):"""

questions = llm.generate(QUESTION_PROMPT.format(title=title, body=body))
question_embeddings = embed.text(questions.split('\n'), model='nomic-embed-text-v1.5', dimensionality=768)

Store the generated questions in the sidecar embeddings file alongside the vectors. Each question maps back to its source document.

LLM models for question generation¶

HyPE question generation is a structured extraction task — it doesn't need frontier-level reasoning. Use the cheapest model that produces well-formed questions.

Default: Use whatever LLM provider is already configured in picobot's config.json. The question generation prompt is simple enough that any model above ~3B parameters produces usable output. Cost estimates assume ~500 input tokens per doc and ~200 output tokens for 5-10 questions.

Don't Chunk¶

At 500-2000 words per KB entry, documents are already coherent semantic units. Chunking yields only 1-4 chunks per doc and provides marginal benefit. Embed the full document alongside the HyPE questions — return the full document on match.

4. No Database: Numpy Brute-Force¶

Architecture¶

Init (2-6 seconds, once):
  1. Load embeddings.npz (or re-embed if stale)
  2. Build in-memory indexes:
     - numpy array for vector search
     - sqlite3 :memory: FTS5 table
     - Python dict for frontmatter

Query (<25ms):
  Frontmatter: tag/title matching
  FTS5: BM25 keyword search (in-memory)
  Vector: numpy cosine similarity (in-memory)
  Fuse: RRF across all three retrievers

Why numpy, not FAISS¶

At 1000 docs with 768-dim vectors:

FAISS is not a replacement for numpy — it's an additional dependency on top of numpy. faiss-cpu is a 15-27 MB compiled C++ package. At 1000 docs, IndexFlatIP is actually slower than raw numpy due to dispatch overhead. FAISS only wins when you use approximate indexes (IVF, HNSW) at 5K+ docs.

# The entire vector search implementation (numpy)
query_vec = embed.text([query], model='nomic-embed-text-v1.5', dimensionality=768)[0]
query_vec = query_vec / np.linalg.norm(query_vec)
scores = embeddings @ query_vec  # cosine similarity (embeddings pre-normalized)
top_k = np.argsort(scores)[-10:][::-1]

Graduate to faiss-cpu IndexFlatIP if KB exceeds 5,000 docs (~10 LOC swap). Graduate to HNSW (faiss IVF or usearch) at 50,000+ docs. Picobot's 100-1000 doc range is firmly in numpy territory.

Memory budget¶

5. Hybrid Search with RRF¶

Three Retrievers¶

Frontmatter retriever (existing) - Exact match on tags, word overlap on title - Returns ranked list of doc IDs - Catches: exact tag matches, title keywords

SQLite FTS5 retriever (replaces current substring search) - BM25 ranking with Porter stemming - Built on in-memory :memory: SQLite database, rebuilt on init (~50-100ms for 1000 docs) - Zero new dependencies (Python stdlib sqlite3) - Catches: keyword matches, partial phrases, stemmed variants

Numpy vector retriever (new) - Cosine similarity on HyPE question embeddings + full-doc embeddings - Catches: semantic matches, intent alignment, synonym bridging

Reciprocal Rank Fusion¶

def rrf_fuse(ranked_lists: list[list[str]], k: int = 60) -> list[str]:
    """Fuse multiple ranked result lists using RRF."""
    scores = {}
    for ranked in ranked_lists:
        for rank, doc_id in enumerate(ranked):
            scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

# Usage
results = rrf_fuse([
    frontmatter_results,
    fts_results,
    vector_results,
])

RRF requires no score normalization, handles different score scales naturally, and is ~15 lines of Python. The k=60 constant prevents any single retriever from dominating.

6. ChromaDB: Upgrade Path, Not Baseline¶

ChromaDB's default embedding function uses ONNX (~17 MB), not PyTorch. Total install is ~150-180 MB with the ONNX model cache. PersistentClient is fully embedded (no server). Cross-collection search is not supported — use a single collection with metadata filtering instead.

Should Picobot Use ChromaDB?¶

Not for the initial implementation. The numpy+FTS5 approach is simpler, lighter (~10 MB vs ~180 MB), and adequate at picobot's scale. ChromaDB adds value when:

• KB exceeds 5,000 docs (HNSW beats brute-force)
• You want persistent indexes that survive process restart without re-embedding
• You want built-in metadata filtering integrated with vector search
• You want automatic embedding (pass text, get vectors)

If ChromaDB is adopted later, use a single collection with topic metadata field, not one collection per topic:

client = chromadb.PersistentClient(path="~/.picobot/data/chroma")
collection = client.get_or_create_collection("knowledge_base")

# Add with metadata
collection.add(
    documents=[body],
    metadatas=[{"topic": "domains", "tags": "python,concurrency", "type": "kb_entry"}],
    ids=[entry_id]
)

# Search with metadata filter
results = collection.query(
    query_texts=[query],
    where={"topic": "domains"},  # optional topic filter
    n_results=10
)

To use Nomic embeddings with ChromaDB instead of the default MiniLM:

from chromadb.utils.embedding_functions import ONNXMiniLM_L6_V2  # default
# OR provide a custom embedding function wrapping nomic

Bloat Caveat¶

ChromaDB ships grpcio, kubernetes, and uvicorn as hard dependencies even for embedded mode (~20 MB wasted). There is no chromadb[lite] option. This is an upstream packaging issue.

7. Unified Retrieval Pipeline¶

Document Type Handling¶

Key Insight: Soul Docs Don't Belong in Search¶

Identity/personality/preference documents should be deterministically injected into every system prompt (as picobot already does with README.md). Retrieval is lossy — search might not return them for a given query, but they're relevant to every query. Keep them out of the vector index entirely.

Weighting with RRF¶

For document types that need boosting, use Weighted Reciprocal Rank Fusion:

def wrrf_fuse(ranked_lists: list[list[str]], weights: list[float], k: int = 60) -> list[str]:
    scores = {}
    for ranked, weight in zip(ranked_lists, weights):
        for rank, doc_id in enumerate(ranked):
            scores[doc_id] = scores.get(doc_id, 0) + weight / (k + rank + 1)
    # Apply type boost
    for doc_id in scores:
        scores[doc_id] *= type_boost.get(doc_type(doc_id), 1.0)
    return sorted(scores, key=scores.get, reverse=True)

type_boost = {
    'kb_entry': 1.0,
    'skill': 1.2,     # skills slightly boosted for capability discovery
    'journal': 0.8,   # journal slightly demoted (raw captures)
}

Recency Decay (Journal Only)¶

Journal entries are raw captures that lose relevance over time. Apply exponential decay:

import math, time

def recency_weight(updated_timestamp: float, half_life_days: float = 30) -> float:
    age_days = (time.time() - updated_timestamp) / 86400
    return math.exp(-0.693 * age_days / half_life_days)  # 0.693 = ln(2)

30-day half-life: a 1-day-old journal entry gets 0.98x, a 30-day-old gets 0.5x, a 90-day-old gets 0.125x. KB entries (stable, processed knowledge) get no decay.

Open Questions¶

1. Nomic API pricing: Sources mention Nomic may have moved to seat-based pricing in 2025. Verify current pricing before relying on remote inference for batch re-embedding.

2. Model load lifecycle: Local Nomic loads in ~1-3 seconds. For the daemon: load once at start, keep in memory. For CLI one-shot: load on demand.

3. Cross-KB search for federated KB: When federated KB arrives, each remote KB would need its own embeddings cache. The numpy approach scales cleanly (concatenate arrays from multiple remotes), but HyPE questions would need to be generated per-remote.

Sources¶

See individual enquiry files for detailed source tables: - enquiry-1.md — Nomic embeddings (7 Tier 1 sources from HuggingFace + Nomic docs) - enquiry-2.md — Frontmatter storage feasibility - enquiry-3.md — What to embed (HyPE, Mem0, Letta, RAPTOR) - enquiry-4.md — FAISS and numpy in-memory search - enquiry-5.md — Hybrid search without a DB (SQLite FTS5, RRF) - enquiry-6.md — ChromaDB as embedded library - enquiry-7.md — Unified retrieval pipeline (Mem0, Letta, weighting)