Skip to content

Tracing & Observability

The tracing module provides full observability for agent executions. Every BaseAgent.execute() invocation produces a trace containing spans for LLM calls, tool invocations, retrieval steps, and grounding decisions. Traces are stored via a pluggable TraceStore interface with in-memory and Firestore implementations. Token usage and cost tracking are built in.

Import

import {
  // Store interface and types
  TraceStore,
  TraceStoreConfig,
  TraceFilter,
  TracePage,
  SessionFilter,
  SessionPage,

  // Data types
  Trace,
  Span,
  SpanKind,
  TraceStatus,
  SpanStatus,
  Session,
  ConfigSnapshot,
  TraceCost,

  // Implementations
  InMemoryTraceStore,
  FirestoreTraceStore,

  // Traced wrappers
  createTracedGeminiClient,
  createTracedToolRegistry,
  createTracedRetrievalService,

  // Cost calculation
  CostCalculator,
  ModelPricing,
  PricingConfig,

  // Export
  exportTracesAsJsonl,
  exportTracesToJsonlString,
} from "@modernpath/agent-framework";

Architecture

graph TD
    Agent["BaseAgent.execute()"] -->|creates| T[Trace]
    Agent -->|records| S[Spans]

    subgraph "Traced Wrappers"
        TG[TracedGeminiClient] -->|emits| LS["LLM Span"]
        TR[TracedToolRegistry] -->|emits| TS["Tool Span"]
        TRS[TracedRetrievalService] -->|emits| RS["Retrieval Span"]
    end

    T --> Store
    S --> Store

    subgraph "TraceStore"
        Store{TraceStore}
        Store -->|dev| IM[InMemoryTraceStore]
        Store -->|prod| FS[FirestoreTraceStore]
    end

    Store -->|export| EX[JSONL Export]
    Store -->|query| UI[Trace UI]
    CC[CostCalculator] -->|attaches| T

Components

TraceStore Interface

The abstract interface for persisting traces, spans, sessions, and config snapshots. Defines the write path (startTrace, endTrace, addSpan, setApiIO), session management (upsertSession), read path (getTrace, listTraces, getSpans), and post-hoc evaluation (setTraceScores, setTraceTags).

InMemoryTraceStore

Map-based implementation for tests and local development. Zero external dependencies. Provides test helper methods for assertions (getAllTraces, getAllSpans, clear).

FirestoreTraceStore

Production-grade Firestore implementation with span write-batching, collection prefixing, and automatic serialization of Date objects to Timestamps. Designed for GCP deployments.

Traced Wrappers

Composition wrappers that intercept GeminiClient, ToolRegistry, and RetrievalService to emit spans automatically. Fire-and-forget: tracing errors never break the primary operation.

CostCalculator

Computes LLM call costs from token counts and model pricing. Ships with built-in Gemini pricing defaults and supports custom pricing via JSON configuration files.

JSONL Export

Exports traces as newline-delimited JSON records for import into LLM eval frameworks (Braintrust, Langfuse, Arize Phoenix) or offline evaluation scripts.

Quick Start

import {
  InMemoryTraceStore,
  createTracedGeminiClient,
  createTracedToolRegistry,
  CostCalculator,
} from "@modernpath/agent-framework";

// 1. Create a trace store
const traceStore = new InMemoryTraceStore({ captureContent: true });

// 2. Wrap your services with traced versions
const tracedGemini = createTracedGeminiClient(geminiClient, {
  getTraceContext: () => currentTraceContext,
  model: "gemini-2.5-flash",
});

const tracedTools = createTracedToolRegistry(toolRegistry, {
  getTraceContext: () => currentTraceContext,
});

// 3. Calculate costs
const costCalc = new CostCalculator();
const cost = costCalc.calculate({
  model: "gemini-2.5-flash",
  inputTokens: 1500,
  outputTokens: 800,
});
console.log(`Cost: $${cost?.cost.toFixed(6)}`);

Trace Data Model

The tracing module uses three primary document types:

Type Description Storage
Trace One BaseAgent.execute() invocation with input, output, status, tokens, and cost traces/{traceId}
Span One sub-operation (LLM call, tool call, retrieval) within a trace traces/{traceId}/spans/{spanId}
Session Groups related traces into a conversation/session for audit listing sessions/{sessionId}
ConfigSnapshot Point-in-time snapshot of agent configuration (prompts, model, schemas) config_snapshots/{snapshotId}