Research Question¶

For an AI agent's knowledge base containing well-structured markdown files (500-2000 words each with YAML frontmatter), what should be embedded for optimal retrieval: full document text, chunks, summaries, or use-case/intent descriptions? How does each strategy perform for intent-based agent queries vs keyword queries?

Executive Summary¶

For picobot's specific context (short, well-structured documents; agent-issued intent-based queries), the highest-impact strategy is hypothetical question embedding (HyPE) -- pre-generating questions each document answers and embedding those instead of the document text. This directly addresses the query-document misalignment problem that is especially acute for intent-based queries. For short documents (500-2000 words), full-document embedding is a viable baseline, and chunking provides marginal benefit at best. A practical hybrid approach would combine HyPE question vectors for retrieval with full-document return for context.

1. Embedding Strategies Compared¶

1.1 Full Document Embedding¶

How it works: Embed the entire document as a single vector. At query time, compare query embedding to document embeddings via cosine similarity.

Strengths: - Simplest implementation; no chunking pipeline needed - Preserves full context -- no information lost to chunk boundaries - For short, single-purpose documents (FAQs, structured KB entries), this is often the optimal approach since the documents are already coherent semantic units - NVIDIA benchmarks found page-level chunking achieved 0.648 accuracy with lowest variance across document types

Weaknesses: - Single vector must represent all topics in a document -- if a document covers multiple concepts, the embedding becomes a diluted average - Longer documents produce embeddings where important details get "washed out" in the mean pooling - Most embedding models have token limits (512-8192 tokens); documents near the upper range may be truncated

Agent query fit: Moderate. An agent query like "find information about error recovery patterns" will match a document about error recovery reasonably well, but may also match documents where error recovery is mentioned briefly alongside other topics.

1.2 Chunk Embedding¶

How it works: Split documents into smaller pieces (typically 128-512 tokens), embed each chunk independently. Return matching chunks or their parent documents.

Strengths: - More precise matching -- each chunk represents a focused semantic unit - Industry standard for large document corpora - Smaller chunks (128-256 tokens) excel at fact-based retrieval; larger chunks (256-512 tokens) better for conceptual queries - RecursiveCharacterTextSplitter at 400-512 tokens achieves 85-90% recall in Chroma benchmarks

Weaknesses: - For short documents (500-2000 words = ~375-1500 tokens), chunking yields only 1-4 chunks -- the overhead of a chunking pipeline may not be justified - Chunks lose surrounding context; chunk boundaries can split coherent ideas - Research shows chunking strategy impact on retrieval is comparable to (and sometimes greater than) embedding model choice -- getting it wrong is costly

Agent query fit: Mixed. Intent-based queries like "I need to research X" are conceptual and broad; they match poorly against narrow, fact-focused chunks. Keyword queries perform better with chunks since specific terms concentrate in specific chunks.

Recommendation for picobot: At 500-2000 words per document, chunking is likely unnecessary. The documents are already near-optimal semantic units. If the KB grows to include longer documents, parent-document retrieval (embed small chunks, return full documents) would be the right approach.

1.3 Summary Embedding¶

How it works: Generate a concise summary of each document, embed the summary. Retrieve documents by matching queries to summary embeddings. Return the full document for LLM context.

Strengths: - Summaries are denser representations that capture the document's core meaning - Reduces noise from tangential content - LangChain's multi-representation indexing uses this: embed summaries for retrieval, return full documents for generation - RAPTOR (Stanford) builds hierarchical summary trees, achieving 20% absolute accuracy improvement on the QuALITY benchmark and new state-of-the-art on QASPER (55.7% F1 with GPT-4)

Weaknesses: - Summaries are lossy -- they discard details, entities, and specific facts - Research shows hierarchical summarization produces increasingly abstract representations that skip entities and numerals - Quality depends entirely on the summarization model; bad summaries degrade retrieval - Requires an LLM call per document at indexing time

Agent query fit: Good for broad conceptual queries ("what do we know about deployment strategies") since summaries capture high-level themes. Poor for specific lookups ("what port does the dashboard use") since those details get summarized away.

1.4 Hypothetical Question Embedding (HyPE)¶

How it works: For each document, use an LLM to generate 5-10 questions the document could answer. Embed these questions (not the document). At query time, match the user's query against question embeddings (question-to-question matching). Return the source document.

Strengths: - Directly solves the query-document misalignment problem: questions and queries have similar linguistic patterns, making semantic matching significantly more accurate - HyPE benchmarks show up to 42 percentage points improvement in context precision and up to 45 percentage points improvement in claim recall vs standard approaches - No runtime LLM overhead -- all question generation happens at indexing time (unlike HyDE which generates a hypothetical answer per query) - Works with any embedding model and is compatible with re-ranking and multi-vector retrieval - Especially powerful for intent-based queries because the generated questions naturally capture intent ("When would I need to handle retries?" matches "I need to implement retry logic")

Weaknesses: - Index size multiplies by the number of questions (5-10x more vectors) - Requires LLM calls during indexing, adding cost and time - Question quality is critical -- poorly generated questions degrade retrieval - Results are non-deterministic across re-indexing runs (different LLM calls produce different questions) - Generated questions must use explicit subjects/objects, not pronouns, to be effective

Agent query fit: Excellent. This is the best strategy for intent-based queries. An agent asking "how do I handle error recovery in the agent loop" will match against pre-generated questions like "What patterns does the agent loop use for error recovery?" -- this is question-to-question matching, which embedding models handle far better than question-to-document matching.

1.5 Use-Case / Intent Description Embedding¶

How it works: For each document, write descriptions of the use cases or intents the document serves, e.g., "As an agent I need to: undertake deep research, validate understanding, fact check." Embed these intent descriptions. Match queries against them.

Strengths: - Conceptually aligned with how agents query -- agents think in terms of tasks and goals - Can be hand-crafted for precision or LLM-generated at scale - Captures the "why would I need this document" rather than "what does this document contain"

Weaknesses: - No established benchmarks or standard implementations (this is a novel idea) - Essentially a variant of hypothetical question embedding where questions are replaced with intent statements - Requires careful design of intent templates; too abstract and they all look alike, too specific and they miss valid queries - Manual curation does not scale; LLM generation has the same issues as HyPE

Agent query fit: Theoretically excellent, but in practice this is HyPE with a different prompt template. The question format ("When would I need retry logic?") is more battle-tested than the intent format ("As an agent I need to: implement retry logic") and produces better embedding differentiation because questions have more varied linguistic structure.

Recommendation: Use HyPE-style question generation as the implementation of this idea. The intent framing is the right mental model, but questions are a more effective embedding format.

2. Comparison Matrix¶

3. How Agent Memory Systems Handle This¶

Mem0¶

Mem0 embeds extracted atomic facts rather than raw conversation text or full documents. An LLM extracts salient facts from conversations ("John is vegetarian", "project deadline is March 15"), and these facts are embedded and stored in a vector database. Retrieval uses semantic similarity on the fact embeddings. This approach achieves 91% lower p95 latency and 90%+ token cost savings vs RAG approaches, with a 10% relative improvement in LLM-as-Judge scores. Mem0 also maintains a graph database for entity relationships alongside the vector store.

Key insight: Mem0 embeds the extracted knowledge, not the source material. This is analogous to summary embedding but at a finer grain -- individual facts rather than document summaries.

Letta (MemGPT)¶

Letta uses a hierarchical memory architecture inspired by operating system memory management: - Core memory: In-context blocks pinned to the LLM context window (always available) - Recall memory: Conversation history searchable via text queries - Archival memory: External embedding-based store, searchable via vector similarity

Archival memory uses standard embedding-based retrieval (embed text passages, search by cosine similarity). The architecture's innovation is not in what gets embedded but in the tiering -- separating always-available memory from search-required memory. Letta explicitly notes that RAG/retrieval is a tool for memory, not memory itself.

A-Mem (Zettelkasten-style)¶

A-Mem creates atomic notes with seven fields: content, timestamp, keywords, tags, context description, embedding, and links. The embedding is computed over the concatenation of all textual fields -- content + keywords + tags + context. This enriched embedding captures more semantic dimensions than content alone. Retrieval uses cosine similarity over these enriched embeddings.

Performance: 45.85% F1 on LoCoMo multi-hop reasoning (GPT-4o-mini) vs 25.52% for MemGPT, while using only ~2,500 tokens vs ~16,900 for baselines.

Key insight: Embedding enriched metadata alongside content (keywords + tags + contextual description) significantly improves retrieval quality. This is directly applicable to picobot's YAML frontmatter.

LangChain Multi-Representation Indexing¶

LangChain's parent document retriever embeds small chunks but returns full parent documents. The multi-vector retriever takes this further: embed summaries (or hypothetical questions, or key concepts) but return full documents. The principle is to use concise, retrieval-optimized representations for matching while preserving full context for generation.

Benchmark comparison from one evaluation: - Multi-Vector Indexing: 85% precision, 82.5% F1 - Parent Document Retrieval: 82% precision, 83.5% F1 - RAPTOR (hierarchical summaries): 88% precision, 87.5% F1

4. ColBERT and Late Interaction Models¶

ColBERT takes a fundamentally different approach: instead of compressing a document into a single vector, it keeps per-token embeddings and computes relevance through token-level interaction (MaxSim). This preserves fine-grained lexical information that single-vector models lose.

Performance: ColBERT outperforms standard bi-encoders significantly while being close to cross-encoder accuracy. Jina-ColBERT-v2 scores 52.6% average across 13 BEIR datasets vs 51.7% for ColBERTv2, with strong results on specific benchmarks (75.0 on TREC-COVID, 79.5 on FEVER).

Storage trade-off: ColBERT stores one vector per token (vs one per document), creating 50-100x more vectors. ColBERTv2 mitigates this with residual compression (6-10x reduction).

Relevance to picobot: ColBERT is interesting but likely overkill for a small KB. Its strength is in large corpora where single-vector representations lose important distinctions. For 500-2000 word documents with good metadata, the simpler approaches (HyPE, enriched embedding) will likely perform comparably with far less complexity.

5. Should Picobot Chunk Its Documents?¶

For documents of 500-2000 words (375-1500 tokens):

No, chunking is probably unnecessary and potentially harmful. Several factors support this:

1. Documents are already semantic units. Picobot KB entries are agent-authored, well-titled, and topically focused. They are not long, multi-topic documents that need decomposition.

2. Chunking at this scale produces 1-4 chunks. The overhead of a chunking pipeline, chunk overlap management, and parent-document linking is not justified for so few chunks.

3. Short, complete documents should not be chunked. Multiple sources confirm that if your data already consists of small, complete pieces of information, chunking can cause problems rather than solve them. The goal is meaningful semantic units -- picobot already has them.

4. Embedding models handle this size well. Modern embedding models (with 512-8192 token context windows) can embed these documents in full without truncation.

Exception: If picobot later ingests long external documents (papers, manuals, codebases), those should be chunked. But for the core KB, embed documents whole.

6. Recommendations for Picobot¶

Primary Strategy: HyPE + Full Document Return¶

1. At indexing time: For each KB document, generate 5-8 hypothetical questions the document answers using an LLM. Store these questions as the retrieval vectors. Keep the full document as the payload returned on match.

2. Enrich the embedding input. Following A-Mem's approach, embed the concatenation of: title + tags + topic + generated questions. This gives the embedding model multiple semantic handles on the document.

3. Embed the full document as a secondary vector. Use a hybrid approach: primary retrieval via question embeddings, with a fallback to full-document cosine similarity. This handles keyword-style queries that HyPE might miss.

4. Do not chunk. At 500-2000 words, the documents are already well-sized semantic units. Embed them whole.

Implementation Sketch¶

For each KB document:
  1. Parse frontmatter (title, tags, topic)
  2. Generate 5-8 hypothetical questions via LLM
  3. Store vectors:
     - N question vectors (primary retrieval)
     - 1 enriched-doc vector: embed(title + tags + topic + content)
  4. All vectors point to the same full document

At query time:
  1. Embed the query
  2. Search across all vectors (questions + enriched docs)
  3. Deduplicate by source document
  4. Return top-k full documents

Question Generation Prompt Template¶

Given this knowledge base document, generate 5-8 questions that an AI agent
might ask when it needs the information in this document. Questions should:
- Be phrased as intent-based queries ("How do I...", "What approach should I use for...")
- Use explicit subjects/objects (no pronouns)
- Cover both the main topic and specific details
- Include at least one broad conceptual question and one specific detail question

Document title: {title}
Document tags: {tags}
Document content: {content}

Why Not Pure Intent Descriptions?¶

The user's original idea of embedding "As an agent I need to: [research, validate, fact-check]" is conceptually sound -- it is essentially HyPE with intent-framed prompts. However, questions produce better embeddings because: - Questions have more varied syntactic structure (who/what/when/how/why) - Intent descriptions tend to converge to similar patterns, reducing discriminability - The question format has extensive research backing (HyPE benchmarks) - Questions naturally capture both broad intent and specific detail

The intent framing is valuable as a mental model for what questions to generate. The prompt template above incorporates this by asking for "questions an AI agent might ask."

Cost Estimate¶

For a KB of ~100 documents at 5-8 questions each: - Indexing: ~500-800 LLM calls (cheap with GPT-4o-mini, ~$0.02 total) - Vector storage: ~600-900 vectors (trivial for any vector DB) - Re-indexing on document update: 1 LLM call per changed document - Runtime: zero LLM overhead (unlike HyDE which calls the LLM per query)

7. Key Sources¶

Research papers and systems referenced: - HyPE (Hypothetical Prompt Embeddings): Up to +42pp context precision, +45pp claim recall vs standard RAG - HyDE (Hypothetical Document Embeddings): Query-time approach; effective but adds per-query LLM latency - RAPTOR (Stanford): Hierarchical summaries; +20% absolute accuracy on QuALITY benchmark - A-Mem: Zettelkasten-style atomic notes; 45.85% F1 vs 25.52% for MemGPT on LoCoMo - Mem0: Extracted-fact embedding; 10% relative improvement over RAG, 91% lower latency - ColBERT: Late interaction; marginal accuracy gain but 50-100x storage - LangChain Multi-Representation: Embed summaries/questions, return full documents