Question¶

How should picobot combine and weight different document types (KB entries, soul/identity docs, skills, workspace files, README.md) in a unified retrieval pipeline? Specifically:

• (a) How do Mem0, Letta/MemGPT, and LangChain handle multi-source retrieval with different document types and importance levels?
• (b) What boosting/weighting strategies exist?
• (c) How should "soul" documents be weighted differently from factual KB entries?
• (d) Should workspace files be included in semantic search or kept separate?
• (e) What is the practical architecture — single index vs. multiple indexes?

Current Picobot Architecture¶

Picobot currently uses a two-tier approach where some content is always injected and other content is retrieved on demand:

Always injected (deterministic, in ContextBuilder.build_system_prompt()): - IDENTITY.md — core agent identity with template variables - Bootstrap files: AGENTS.md, SOUL.md, USER.md, TOOLS.md - knowledge/README.md — root KB context summary - TODO.md — pending tasks (if non-empty) - Always-loaded skills (skills with always: true in metadata) - Skills summary (names + descriptions of all available skills)

Retrieved on demand (via kb tool, keyword-based): - KB entries across self/, user/, projects/, domains/, journal/ topics - Two-phase search: frontmatter tag/title matching, then full-text substring fallback - No embeddings, no scoring beyond term overlap ratio

Not searched at all: - Per-topic README.md files (templates, not content) - SCRATCH.md (async messaging channel) - Skill SKILL.md files (loaded by read_file, not searched) - Session state, logs, traces

The gap: there is no unified retrieval that spans document types, no semantic search, no weighting by document importance, and no way to find relevant skills or workspace context based on the user's query.

(a) How Existing Systems Handle Multi-Source Retrieval¶

Mem0¶

Mem0 (arxiv 2504.19413) organizes memory into four scoped layers with a clear retrieval hierarchy:

Retrieval pipeline: The search pipeline pulls from all layers, ranking user memories first, then session notes, then raw conversation history. This is a fixed priority ordering, not a learned weight — the assumption is that stable personal facts are more valuable than transient session context.

Extraction and consolidation: New messages are processed by an LLM that extracts candidate facts, then each fact is compared against the top-10 semantically similar existing memories. The LLM decides one of four operations: ADD, UPDATE, DELETE, or NOOP. This means Mem0 actively manages memory quality rather than just accumulating entries.

Graph variant (Mem0g): The graph-enhanced version uses dual retrieval — entity-centric (find entity nodes, traverse relationships) and semantic triplet (embed the query, match against textual encodings of relationship triplets above a relevance threshold). This doubles token usage (~14k vs ~7k per conversation) but excels at relational queries.

Key insight for picobot: Mem0's layered priority (user > session > conversation) maps well to picobot's document types. Identity/soul docs are analogous to organizational memory (always relevant), KB entries to user memory (stable facts), and workspace files to session memory (task-specific, ephemeral).

Letta / MemGPT¶

Letta implements an OS-inspired memory hierarchy with three tiers:

Core memory blocks are the critical design pattern. Each block has a label, description, value, and character limit. The description field is what the agent uses to decide which block to read/write — it acts as a schema definition. Core blocks are always visible in the context window with zero retrieval latency.

Archival memory is the only tier that uses semantic search. The agent must explicitly invoke a search tool to query it, and results are injected into the current context. This is the equivalent of RAG over a personal knowledge base.

Priority by design, not by scoring: Letta does not numerically weight memory types. Instead, it uses an architectural priority: core memory is always present (highest priority), archival memory requires an explicit search (medium priority), and recall memory requires a different explicit search (lowest priority). The agent itself decides when to retrieve based on its assessment of whether it has enough context.

Key insight for picobot: The core memory pattern validates picobot's approach of always injecting IDENTITY.md, SOUL.md, and TODO.md. These are the agent's "core memory blocks." The question is not whether to include them in semantic search — they should remain deterministically injected. Semantic search is for the archival tier (KB entries, skills, old workspace context).

LangChain EnsembleRetriever¶

LangChain takes a more mechanical, composable approach with the EnsembleRetriever:

• Accepts a list of retriever objects (e.g., BM25 retriever, vector store retriever, metadata-filtered retriever)
• Each retriever has a corresponding weight (float)
• Results are merged using Weighted Reciprocal Rank Fusion (WRRF)

The formula for each document's fused score:

score(d) = sum over all retrievers r:  weight_r * (1 / (rank_r(d) + k))

Where k is a constant (typically 60) that dampens the impact of rank position.

This is a rank-based fusion, not a score-based fusion — it only uses the ordering from each retriever, not the raw similarity scores. This makes it robust across retrievers with different score scales.

Key limitation: LangChain's EnsembleRetriever does not natively support document-type-specific weighting within a single retriever. To boost KB entries over skills, you would need to either (a) use separate retrievers with different weights, or (b) apply a post-retrieval reranking step.

Key insight for picobot: WRRF is the right fusion algorithm when combining multiple retrieval sources (e.g., a KB retriever and a skills retriever). But document type boosting needs to happen either at the retriever level (separate retrievers per type) or as a post-retrieval score adjustment.

(b) Boosting and Weighting Strategies¶

Strategy 1: Multiplicative Document Type Boost¶

Apply a multiplier to the raw similarity score based on document type metadata:

final_score = similarity_score * type_boost[doc.type]

Example type boosts for picobot:

Tradeoff: Multiplicative boosting preserves the relative ordering within a document type (a highly relevant journal entry still beats a weakly relevant self/ entry). The boost values are relative, not absolute — a 1.3x boost on a 0.3 similarity score (= 0.39) will not beat a 1.0x boost on a 0.5 similarity score (= 0.5).

When to use: When you want to express "all else being equal, prefer this type."

Strategy 2: Additive Document Type Boost¶

Add a fixed bonus to the score based on document type:

final_score = similarity_score + type_bonus[doc.type]

Tradeoff: Additive boosting has a stronger effect on low-scoring documents. A +0.1 bonus on a document with 0.2 similarity (= 0.3) is a 50% relative increase, but on a document with 0.8 similarity (= 0.9) it is only a 12.5% increase. This means additive boosting can promote marginally relevant documents of a preferred type above genuinely relevant documents of a non-preferred type.

When to use: When you want to guarantee a minimum number of results from a specific type, regardless of relevance. Generally not recommended for picobot's use case.

Strategy 3: Recency Decay¶

Apply a time-based decay to score, biasing toward recently updated documents:

recency_factor = exp(-lambda * days_since_update)
final_score = similarity_score * recency_factor

Or use bucketed weights (as Ragie does):

When to use: For journal entries and workspace files where recent context is usually more relevant. Not appropriate for identity documents or stable domain knowledge — those should be recency-neutral.

Picobot-specific approach: Apply recency decay only to journal/ entries and workspace files. KB entries in self/, user/, projects/, domains/ should use their updated frontmatter timestamp only as a tiebreaker, not as a primary scoring factor.

Strategy 4: Access Frequency Boost¶

Track how often documents are retrieved and read, then boost frequently-accessed documents:

frequency_factor = log(1 + access_count) / log(1 + max_access_count)
final_score = similarity_score * (1 + alpha * frequency_factor)

Where alpha controls the strength of the frequency boost (e.g., 0.2 for a mild 20% max boost).

When to use: For surfacing "working set" documents — things the agent or user is actively referencing. This creates a positive feedback loop, so it needs a decay mechanism (e.g., exponential decay on access counts over time).

For picobot: This is a phase 2 optimization. Track access counts in frontmatter (accessed: 14, last_accessed: 2025-01-15) but do not implement frequency boosting until the base retrieval pipeline is stable.

Strategy 5: Weighted Reciprocal Rank Fusion (WRRF)¶

When combining results from multiple retrieval sources (not just multiple document types within a single index), use WRRF:

score(d) = sum over retrievers r: weight_r / (k + rank_r(d))

With k = 60 as the standard constant. The weight per retriever controls the relative influence of each source.

Example configuration for picobot:

Key advantage: WRRF is rank-based, so it handles the different score scales from different retrievers gracefully. A cosine similarity of 0.85 and a BM25 score of 12.4 cannot be directly compared, but their ranks within each retriever's results can.

Recommended Approach for Picobot¶

Combine strategies 1, 3, and 5:

1. WRRF to merge results from multiple retrieval sources (vector, keyword, skills)
2. Multiplicative type boost as a post-fusion adjustment
3. Recency decay applied selectively to journal/ and workspace documents only

The combined formula:

raw_score = WRRF(vector_rank, keyword_rank, skills_rank)
type_adjusted = raw_score * type_boost[doc.type]
final_score = type_adjusted * recency_factor(doc)  # only for journal/workspace

(c) Weighting Soul/Identity Documents vs. Factual KB Entries¶

The Core Principle: Soul Documents Should Not Be Retrieved¶

The key insight from both Letta and Mem0 is that identity/soul documents operate at a fundamentally different level than factual KB entries. They should be deterministically injected, not semantically retrieved.

Why: 1. Identity is always relevant. The agent's personality, constraints, and behavioral guidelines apply to every interaction. Requiring a relevance match to surface them is a design error. 2. Retrieval is lossy. If the soul document says "always be concise" but the user asks about Python decorators, a semantic search for "Python decorators" will never surface the conciseness instruction. Yet it is still needed. 3. Stability vs. dynamism. Soul documents change rarely (perhaps once per week or month). KB entries change frequently. Mixing them in the same retrieval pipeline conflates two different information lifecycles.

Picobot's Current Approach is Correct¶

Picobot already deterministically injects IDENTITY.md, SOUL.md, USER.md, and TOOLS.md via the bootstrap file mechanism. This is architecturally sound and matches the Letta core memory pattern.

What Should Change¶

The root knowledge/README.md is also always injected, which is correct — it serves as a navigational aid for the KB, not as a factual document. However, as the KB grows, the README should contain only structural metadata (topic descriptions, entry counts, last-updated dates), not summaries of actual content. Content summaries belong in per-topic READMEs or in a generated index.

Per-topic README.md files (e.g., self/README.md) currently contain write templates. They should be extended with a "Topic Summary" section that is periodically regenerated (e.g., at session boundaries) to summarize the key entries in that topic. These summaries become the "core memory" equivalent for each knowledge domain — always injected into the system prompt, providing the agent with an awareness of what it knows without requiring a search.

Tier Model for Picobot Documents¶

(d) Workspace Files: Semantic Search or Deterministic Injection?¶

Files That Should Remain Deterministically Injected¶

TODO.md: Always injected (picobot already does this). The agent needs to see pending tasks on every turn to maintain task continuity. Semantic search would miss the structural meaning of a TODO list (checkboxes, ordering, priority) and would only match on keyword content.

Always-loaded skills: Already handled by the always: true metadata flag. These define capabilities the agent should always be aware of.

Files That Need Special Treatment¶

SCRATCH.md: This is an async messaging channel, not a knowledge document. It has three sections with different semantics: - "To Claude" — instructions from the human, should be injected at session start (already specified in CLAUDE.md behavior) - "To Human" — status updates from the agent, should be writeable but not searchable - "Notes" — shared scratchpad, could be keyword-searchable but not embedded

Recommendation: SCRATCH.md should NOT be in the semantic search index. Its "To Claude" section should be injected at session start (read, act, clear). Its "Notes" section could be searched via keyword match only, as a fallback source when KB search returns insufficient results.

Files That Should Be Searchable¶

SKILL.md files: These should be in the retrieval index. When the user asks "can you browse the web?" or "help me schedule a task," the retrieval pipeline should surface the relevant skill. Currently, skills are only discoverable via the summary list in the system prompt. Adding them to semantic search would allow the agent to discover relevant skills based on query context.

KB entries (all topics): Already the primary search target. Adding embeddings improves recall significantly over the current keyword-only approach.

Files That Should NOT Be Searchable¶

• Session logs (~/.picobot/data/logs/*.jsonl) — structured data, not natural language
• Traces (~/.picobot/data/traces/traces.jsonl) — debugging artifacts
• Sub-agent registry (~/.picobot/data/subagents/registry.json) — structured data
• Cron jobs (~/.picobot/data/cron/jobs.json) — structured data
• Per-topic README.md write templates — structural, not content

(e) Architecture: Single Index vs. Multiple Indexes¶

Option A: Single Index with Metadata-Based Type Filtering¶

Store all searchable documents (KB entries + skills) in a single vector index. Each document gets metadata:

{
  "type": "kb_entry",
  "topic": "projects",
  "title": "Picobot Architecture Decisions",
  "tags": ["architecture", "decisions"],
  "created": "2025-01-15",
  "updated": "2025-01-20",
  "source_path": "projects/picobot-architecture.md"
}

Advantages: - Simpler implementation — one index, one embedding model, one search call - Cross-type discovery: a query about "web browsing" can surface both a KB entry about browser usage patterns and the browser SKILL.md - Metadata filtering enables scoped searches: type=kb_entry AND topic=projects - Easier to maintain consistency (one indexing pipeline)

Disadvantages: - Different document types may have different optimal chunk sizes (skills are short and self-contained; KB entries vary widely) - Type boosting must be applied as a post-retrieval score adjustment, not as part of the vector search itself - Index size grows linearly with all document types combined

Option B: Multiple Indexes with Result Merging¶

Separate indexes per document type, merged via WRRF:

• KB index: All KB entries across topics, chunked as needed
• Skills index: All SKILL.md files, stored as whole documents (they are typically short)
• Workspace index: TODO.md, SCRATCH.md notes section (keyword-only, no embeddings)

Advantages: - Per-type tuning: different chunk sizes, different embedding models, different search strategies - Type weighting is built into the architecture (WRRF weights per retriever) - Can mix retrieval strategies: vector search for KB, keyword search for workspace files - Can independently update/rebuild each index

Disadvantages: - More complex implementation: multiple indexes, multiple search calls, fusion logic - Higher latency: parallel searches + fusion step - Risk of result fragmentation: relevant context split across indexes may not be surfaced together

Option C: Single Index + Retrieval Router (Recommended)¶

A hybrid that gets the benefits of both:

1. Single vector index for all embeddable documents (KB entries + skills)
2. Keyword fallback for non-embeddable documents (workspace files, SCRATCH.md notes)
3. Retrieval router that decides which sources to query based on the query type

The router is simple rule-based logic, not an LLM call:

def route_query(query: str, context: dict) -> list[str]:
    sources = ["kb_vector"]  # always search KB embeddings

    # Add keyword search as supplement
    sources.append("kb_keyword")

    # Search skills if query mentions capabilities or "how to"
    if has_capability_intent(query):
        sources.append("skills_vector")

    # Search workspace if query mentions tasks, TODOs, or current work
    if has_task_intent(query):
        sources.append("workspace_keyword")

    return sources

Results from all activated sources are merged via WRRF, then type-boosted, then returned.

Why this is best for picobot: - Keeps the common case fast (just KB vector search for most queries) - Skills and workspace are only searched when contextually relevant - Single index for the primary use case avoids the complexity of multi-index management - The router is deterministic and cheap (no LLM call) - Metadata filtering on the single index handles topic scoping

Recommended Index Schema¶

For the single vector index (e.g., using ChromaDB, which picobot may adopt per other enquiries):

collection.add(
    ids=["kb::projects/architecture.md::chunk-0"],
    documents=["chunk text here..."],
    embeddings=[...],  # or let Chroma compute them
    metadatas=[{
        "doc_type": "kb_entry",       # kb_entry | skill | workspace
        "topic": "projects",           # KB topic, or "skills", or "workspace"
        "title": "Architecture Decisions",
        "tags": "architecture,decisions",
        "source_path": "projects/architecture.md",
        "created": "2025-01-15T10:00:00",
        "updated": "2025-01-20T14:30:00",
        "chunk_index": 0,
        "total_chunks": 3,
    }]
)

Retrieval Flow¶

User query
    |
    v
[Retrieval Router] -- determines which sources to activate
    |
    +---> [KB Vector Search] --> top-k results with scores
    |
    +---> [KB Keyword Search (BM25/current)] --> top-k results with ranks
    |
    +---> [Skills Vector Search] (if activated) --> top-k results
    |
    +---> [Workspace Keyword Search] (if activated) --> top-k results
    |
    v
[Weighted Reciprocal Rank Fusion] -- merge all result lists
    |
    v
[Type Boost] -- multiply scores by doc_type weight
    |
    v
[Recency Decay] -- apply to journal/ and workspace docs only
    |
    v
[Deduplicate + Top-N] -- return final results

Summary of Recommendations¶

1. Do not search soul/identity documents. Keep IDENTITY.md, SOUL.md, USER.md, TOOLS.md as deterministic injections. This matches the Letta core memory pattern and Mem0's organizational memory tier.

2. Use a single vector index for KB entries and skills, with metadata-based type filtering and topic scoping.

3. Use WRRF to merge results from vector search, keyword search, and (when activated) skills search and workspace keyword search.

4. Apply multiplicative type boosting as a post-fusion score adjustment. Start with conservative boosts (self/ = 1.2, user/ = 1.1, journal/ = 0.8, skills = 0.9) and tune based on retrieval quality.

5. Apply recency decay only to journal/ and workspace files. Use exponential decay or bucketed weights. Do not apply recency bias to stable knowledge in self/, user/, projects/, domains/.

6. Keep workspace files (TODO.md, SCRATCH.md) out of the vector index. TODO.md is always injected. SCRATCH.md notes can be keyword-searched as a fallback. Session state, logs, and traces are never searched.

7. Add SKILL.md files to the vector index with doc_type: skill metadata. This enables capability discovery through semantic search.

8. Extend per-topic README.md files with auto-generated topic summaries. Inject these summaries into the system prompt alongside the root README to give the agent awareness of its knowledge without requiring a search.

9. Implement a simple retrieval router that activates skills search for capability-related queries and workspace search for task-related queries, keeping the common-case fast path as KB-only vector search.

10. Defer access frequency boosting to a later phase. Track access counts now (in frontmatter metadata) but do not use them for scoring until the base pipeline is validated.

Sources¶

• Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (arXiv 2504.19413)
• Mem0 Memory Types Documentation
• Mem0 Context Engineering Guide
• Letta/MemGPT: Intro to Letta
• Letta Memory Blocks Documentation
• Letta Agent Memory Blog Post
• Letta Memory Management
• LangChain EnsembleRetriever Documentation
• Weighted Reciprocal Rank Fusion in Elasticsearch
• Reciprocal Rank Fusion in OpenSearch
• How To Prioritize Documents In RAG Retrieval Systems
• Ragie Retrievals: Recency Bias
• Solving Freshness in RAG (arXiv 2509.19376)
• Azure AI Search Scoring Profiles
• Additive vs Multiplicative Boosting (Solr & Elasticsearch)
• RAG Techniques Repository (NirDiamant)
• Guide to Multi-Index Retrieval for Optimized RAG
• Context Engineering for Agents (Lance Martin)
• LangChain: How Agents Can Use Filesystems for Context Engineering
• Context Engineering for Coding Agents (Martin Fowler)