Build Note 02: A System That Knows What It Knows

In Version Control for Beliefs, we made the case that AI memory needs the same engineering discipline we’ve given code: versioned beliefs, provenance tracking, regression detection, and accountability trails.

This is the build note on how we’re actually implementing that.

If you haven’t read the companion piece, the short version: most AI memory systems optimize for recall — did the system find something relevant? We’re building a system optimized for not being wrong. That’s a different problem with different engineering requirements.

Where We Started — and Why It Wasn’t Enough

We’ve been running a three-layer memory system for months: short-term conversational context, medium-term session recall, and long-term persistent storage backed by vector embeddings. It works. Conversations carry context. Sessions build on prior sessions.

But as our usage expanded across payment analytics, infrastructure management, client operations, internal strategy, and personal productivity, the failure modes became predictable.

A query about one client’s payment setup surfaced context from an unrelated internal discussion. The system retrieved a recommendation it had made three months prior — but the underlying data had changed and the recommendation was now wrong, and the system had no mechanism to know that. Two contradictory facts coexisted in memory with no resolution.

These aren’t edge cases. They’re the natural consequence of treating memory as flat retrieval. When everything is stored with equal weight and equal validity, the system’s confidence in its answers has no relationship to the actual reliability of its sources.

Architectural Foundations

Two systems shaped our thinking, though we’ve diverged significantly from both.

Hindsight, from Vectorize.io, introduced epistemic separation — organizing memory into distinct networks for facts, experiences, observations, and opinions, with a retain-recall-reflect loop. The key insight: separating what the system has observed from what it has inferred changes the reliability of everything downstream.

Ori-Mnemos contributed a structural model for memory decay using ACT-R base-level learning equations from cognitive science — memories fade based on usage, not arbitrary timers — and Louvain community detection for identifying natural clusters in the memory graph.

Both are excellent. Neither solves the problem we have: running a consultancy across multiple sensitive domains where the system needs to be demonstrably right, needs to know when it isn’t, and needs to keep client data architecturally isolated.

The Object Model: Claims, Not Conversations

The core design decision is that we don’t store conversations. We extract structured claims — discrete, attributable statements with explicit confidence scores, temporal context, entity references, and links to their source artifacts.

A claim knows where it came from (origin artifact), when it was created, how confident the system is in its accuracy, what entity it relates to, what memory class it belongs to (factual, procedural, preference, contextual), and when it expires.

This is the atomic unit. The broker, the lifecycle engine, the evaluation harness — everything operates on claims. Not on conversation logs. Not on retrieved text chunks. On structured, inspectable, attributable units of belief.

Supersession Chains: Git Log for Beliefs

When new information contradicts an existing claim, the old claim doesn’t get deleted or overwritten. It gets linked.

The new claim carries a supersedes reference to the old one. The old claim carries a superseded_by reference forward. The full chain is walkable — you can trace how the system’s understanding of any topic evolved through every version of every belief.

Superseded claims are excluded from retrieval by default. They’re still in the system — queryable for audit purposes, available when include_superseded: true is explicitly requested. The system remembers what it used to believe while being clear about what it currently believes. That distinction sounds simple. Almost nobody builds it.

The practical value: when a client asks “why did your system recommend X six months ago?” we can walk the supersession chain and reconstruct exactly what the system believed at that point, what changed, and when. That’s the automated equivalent of git blame — traceable accountability for every belief the system has held.

TTL and Recall Gating: Automated Deprecation

Claims have time-to-live values determined by their type and memory class. A factual claim about a client’s current processing volume expires faster than a procedural claim about how interchange rates are structured. An observation about a market trend has a different shelf life than a recorded client preference.

When a claim reaches its TTL, it doesn’t vanish. It becomes ineligible for recall — the eligible_for_recall flag flips to false. The broker won’t serve it as current information, but it persists in the system for audit, for supersession chain integrity, and for pattern analysis.

The system forgets on purpose. On schedule. With a complete record of what it forgot and why. Stale information doesn’t silently persist and doesn’t silently disappear. It ages through a mechanism that’s inspectable.

Coverage Classification: Reporting What You Don’t Know

Every query processed by the broker gets classified: covered, partial, or uncovered.

Covered means the system has high-confidence claims relevant to the query. Partial means some relevant claims exist but coverage is incomplete. Uncovered means the system has no reliable information.

This is the feature that sounds trivially obvious and almost nobody implements. Most systems return their best guess and let confidence scores create an illusion of calibration. Ours treats “I don’t have reliable information on this” as a first-class response — not a fallback, but a designed output.

For multi-agent architectures, this is critical. When Agent B receives output from Agent A, coverage classification tells it whether it’s working with verified ground truth or generated prose. That’s the circuit breaker that prevents hallucination from cascading through an agent chain.

Semantic Dedup at Ingestion

Before a new claim enters the system, it’s compared against existing claims for the same entity and memory class using cosine similarity. Near-identical claims (above 0.95 similarity threshold) get caught and merged rather than accumulating as noise.

For the pilot, we’re using local embeddings — sentence-transformers running on CPU. No API dependency, no data leaving the local environment, good enough for dedup at pilot scale. Brute-force cosine similarity against existing claims for the same entity. If scale increases, we add an approximate nearest neighbor index. But we’re not engineering for scale we don’t have yet.

The system stays lean because it’s designed to resist bloat architecturally, not manage it after the fact.

The Louvain Layer — Strictly Read-Only

We’re using Louvain community detection to visualize how claims cluster naturally around domains. The clusters reveal things invisible from inside any single query: orphan nodes belonging to no community, bridge nodes connecting domains, competency gaps where coverage is thin.

The constraint: this layer is read-only. It cannot influence retrieval. It cannot influence ranking. It cannot affect what the system treats as true.

This is an architectural boundary. The temptation to let clustering feed back into retrieval is enormous — “the system knows these topics are related, so surface related context automatically.” That’s exactly the kind of uncontrolled synthesis that produces plausible answers the system can’t justify. The visual layer observes. It does not intervene.

Strictly separated from the canonical memory store and the broker. No premature merging. This is a principle we enforce at the infrastructure level: if the visualization layer can influence truth, it eventually will — so we’ve made it structurally impossible.

PII Locality and the Sanitization Frontier

Personal and client-identifiable information stays local by default. Embeddings can be generated and matched without raw data transiting to external services. This is architectural — the system physically can’t leak what it doesn’t send.

The active development frontier is a sanitization layer for corporate deployment. As we extend this system beyond internal use, we need a pipeline that strips or anonymizes sensitive data before it enters any shared memory tier. The goal: persistent memory — continuity, institutional knowledge, pattern recognition — in environments where data governance isn’t optional. You shouldn’t have to choose between a system that remembers and a system that’s safe.

The Evaluation Framework

We’re piloting this with quantitative stop-point criteria at every gate. Some of the targets:

Entity extraction error rate below 30%. Claim noise rate below 20%. Retrieval relevance above 70% on human-judged test queries. And stale hallucination tolerance: zero percent.

Zero. Not because we believe we’ll achieve it permanently, but because making the target explicit changes what you engineer for. When the acceptable stale hallucination rate is anything above zero, it becomes a rounding error that nobody prioritizes. When it’s zero, the architecture has to actively prevent it.

We built the evaluation harness before the system — a frozen test set of queries authored before implementation, so we can’t unconsciously tune the system to its own test data. Baseline scores from the current memory system are recorded. Graph broker scores will be measured against the same queries. Structured comparison, not vibes.

We’re also using our own AI tooling to audit the architecture. Our internal agent reviewed the build plan and produced a structured concern document — ten items, severity-ranked, each with a proposed resolution and a decision gate. It caught that the evaluation harness was defined as a deliverable but unspecified as tooling. It caught that the persistence technology was undefined and blocking step one. It caught that stop-point criteria were qualitative where they needed to be quantitative.

Building tools that audit the building of tools is recursive. It’s also measurably better engineering.

Phased Data Expansion

We’re not loading everything at once.

Phase A: last 90 days of trusted sources only. Phase B: historical data in validated batches, with full evaluation after each batch. Phase C: new sources, only after stability is confirmed.

After every expansion, we re-run: coverage reports, precision metrics, negative case validation, stale hallucination checks, boundary tests, entity and alias growth rates, cluster quality, and orphan rate. If metrics degrade, we stop and diagnose before adding more data.

Scale carefully. Observe behavior. Then optimize. Not the other way around.

Where This Is Heading

This system will manage our company RAG — evolving from git-versioned static documents to dynamic, context-aware, temporally-grounded retrieval. It will feed into telOS Mission Control alongside Prometheus telemetry, Linear project data, and processor APIs. And it will serve as the epistemic foundation for any multi-agent capabilities we deploy — the shared ground truth layer that makes agent handoffs trustworthy rather than hopeful.

Getting memory right isn’t a feature. It’s the infrastructure that makes every other AI capability reliable or unreliable. If the system can’t be right, can’t tell you when it’s uncertain, can’t track how its understanding changed, and can’t contain the blast radius of a single hallucination — then everything built on top inherits those failures.

The Nugget: “Getting memory right isn’t a feature. It’s the infrastructure that makes every other AI capability reliable or unreliable.”

We’d rather get the roots right than ship something impressive that regresses.

---

For the broader argument on why AI memory needs version control discipline, read Version Control for Beliefs →

This is part of our Build Notes series — technical logs from the workshop at Telos One.

Want to follow the build? Subscribe to Digital Nuggets →