Skip to content

Knowledge Base

The Knowledge Base module implements a complete RAG (Retrieval-Augmented Generation) pipeline for grounding agent responses in authoritative documents. It spans the full lifecycle: document ingestion, semantic retrieval, canonical document resolution, document composition, and configurable grounding policies.

Pipeline Overview

flowchart LR
    subgraph Ingest
        A[Documents] --> B[IngestionService]
        B --> C[File Search Store]
    end

    subgraph Retrieve
        D[User Query] --> E[RetrievalService]
        E --> C
        E --> F["KnowledgeAnswer\n(text + sources)"]
    end

    subgraph Ground
        F --> G[CanonicalGroundingService]
        G --> H["Fetch full documents\nfrom S3/SharePoint/GCS"]
        H --> I[CompositionLoader]
        I --> J["Expanded bundle\n(root + includes)"]
    end

    subgraph Policy
        J --> K["decideGrounding()"]
        F --> K
        K --> L["GroundingDecision\n(prompt text + attachments)"]
    end

    L --> M[LLM Synthesis]

Module Overview

Service Purpose Page
IngestionService Upload documents into Gemini File Search Stores with metadata. Ingestion
RetrievalService Semantic search with store name resolution, caching, and source extraction. Retrieval
CanonicalGroundingService Resolve RAG sources to full canonical documents from ground-truth storage (S3, SharePoint, GCS). Canonical Grounding
CompositionLoader Expand markdown documents with includes[] frontmatter into complete bundles. Document Composition
GroundingPolicy / decideGrounding() Control what grounding inputs (RAG chunks vs. full documents) reach the LLM. Grounding Policy

How It Works

1. Ingestion

Documents are uploaded to Gemini File Search Stores with customMetadata that records the document's canonical source location (S3 bucket/key, SharePoint site/item, or GCS bucket/object). This metadata enables later resolution from RAG search results back to the full ground-truth document.

const ingestion = new IngestionService(stores);
await ingestion.ingestBuffer(pdfBuffer, "cold-complaint-sop.pdf", "hvac-kb", {
  metadata: [
    { key: "source_type", string_value: "s3" },
    { key: "s3_bucket", string_value: "acme-sops" },
    { key: "s3_key", string_value: "sops/cold-complaint.pdf" },
  ],
});

2. Retrieval

At query time, the RetrievalService performs a semantic search against the store and extracts structured source metadata from the grounding response.

const retrieval = new RetrievalService(stores);
const answer = await retrieval.search("How do I handle a cold complaint?", "hvac-kb");
// answer.text     -> synthesized answer
// answer.sources  -> [{ documentName, documentId, chunk, score, storeName }]

3. Canonical Grounding

The CanonicalGroundingService takes the RAG sources, resolves their customMetadata to canonical document pointers, fetches the full documents from the original storage, and optionally expands markdown documents through their includes[] frontmatter.

const grounding = new CanonicalGroundingService(stores, fetcher, {
  maxDocuments: 3,
  compositionPolicy: { maxDepth: 2, maxDocs: 8 },
});
const result = await grounding.prepareFromSources(answer.sources, ctx);
// result.allDocumentsForAttachment -> PreparedDocument[] ready for Gemini

4. Grounding Policy

The decideGrounding() function applies a configurable policy to determine what reaches the LLM prompt:

const decision = decideGrounding({
  retrievedText: answer.text,
  preparedDocuments: result.allDocumentsForAttachment,
  policy: { mode: "hybrid" },
});
// decision.retrievedTextForPrompt  -> RAG text for the prompt (or "")
// decision.includeAttachments      -> whether to attach full documents

Grounding Modes at a Glance

Mode RAG Chunks in Prompt Full Documents Attached
chunks_only Yes No
attachments_only No (graceful fallback if no attachments) Yes
hybrid Configurable Yes

See Grounding Policy for full details on each mode.

Supported Storage Backends

Backend Pointer Type Required Package
Amazon S3 / MinIO S3DocumentPointer @aws-sdk/client-s3
Google Cloud Storage GcsDocumentPointer @google-cloud/storage
SharePoint SharePointDocumentPointer Built-in (uses Graph API)

Storage backends are loaded dynamically -- only install the SDK for the backend you use.

Debug Logging

Each service supports granular debug logging via the DEBUG environment variable:

# Individual modules
DEBUG=retrieval-service node app.js
DEBUG=canonical-grounding node app.js
DEBUG=composition-loader node app.js

# All knowledge base modules
DEBUG=retrieval-service,canonical-grounding,composition-loader node app.js

# Everything
DEBUG=* node app.js