09 / 20 Memory System src/memdir/ · src/services/autoDream/ · src/services/extractMemories/

MEMORY.md is a pointer index.

It's NOT a memory store — it's an index of one-line pointers to topic files. Truncated at 200 lines / 25KB. Topic files are auto-surfaced by Sonnet (not FileReadTool). Session transcripts are grep-only. Background consolidation via autoDream fires after 24h + 5 sessions.

200
Max lines in MEMORY.md (pointer index)
25KB
Max bytes — truncated at line AND byte boundary
5
Topic files auto-surfaced by Sonnet per turn
4
Memory types — user, feedback, project, reference

MEMORY.md — The Pointer Index

MEMORY.md is always injected into the system prompt. But it contains no actual memory content — just one-line pointers to topic files.

What MEMORY.md looks like

- [User role](user_role.md) — data scientist focused on observability
- [Testing preferences](feedback_testing.md) — use real DB, not mocks
- [Merge freeze](project_freeze.md) — starts 2026-03-05 for mobile release
- [Bug tracker](reference_linear.md) — pipeline bugs in Linear "INGEST"

Each line: - [Title](file.md) — one-line hook
Max ~150 chars per line. Lines after 200 truncated. Warning appended if truncated.

Two-step save process

  1. Step 1: Write memory to its own file (e.g., feedback_testing.md) with YAML frontmatter: name, description, type
  2. Step 2: Add a one-line pointer to MEMORY.md

Never write memory content directly into MEMORY.md. It's an index, not a store.

Truncation: Enforced at both line (200) AND byte (25,000) boundaries. Line truncate first, then byte-truncate at last newline before 25KB. If truncated, warning appended: "Only part of it was loaded. Keep entries under ~150 chars; move detail into topic files."

3-Layer Storage Architecture

Write Paths
~/.claude/projects/.../memory/
Read Paths
Manual write
User request
MEMORY.md
Layer 1 — always in system prompt
System prompt
Injected every turn
→
← always
autoDream
Background (24h+5 sessions)
Topic files (*.md)
Layer 2 — Sonnet-selected
Sonnet selection
Up to 5, as attachments
- - →
← if relevant
extractMemories
Per-turn (forked agent)
Transcripts (.jsonl)
Layer 3 — grep-only, never fully read
Targeted grep
Narrow terms only
- - →
← grep only
consolidationLock.ts — PID-based race protection, 60min stale guard

How Sonnet Selects Relevant Memories

Not FileReadTool — an automatic background system using Sonnet as a relevance ranker.

1
Scan memory directory
memoryScan.ts reads up to 200 .md files (excluding MEMORY.md). Extracts: filename, mtime, frontmatter description (first 30 lines), type. Sorted newest first.
2
Sonnet side-query with formatted manifest
Calls sideQuery() with Sonnet model. Input: user's current query + formatted manifest ([type] filename (timestamp): description). System prompt: "Select memories you are CERTAIN will be helpful. Filter out recently-used tool docs."
3
JSON response → up to 5 files
Sonnet returns { selected_memories: string[] } (max 256 tokens). Already-surfaced memories filtered out. Results: up to 5 {path, mtimeMs} entries.
4
Injected as attachments (not tool calls)
readMemoriesForSurfacing() loads file contents. Injected as relevant_memories attachments — the model sees them in context without any FileReadTool call. Fully automatic.

4 Memory Types

user

Role, goals, knowledge, preferences. "User is a data scientist focused on observability." Tailors responses to expertise level.

feedback

What to do/avoid based on corrections AND confirmations. "Use real DB, not mocks — burned last quarter." Includes Why + How to apply.

project

Ongoing work, goals, deadlines, incidents. "Merge freeze starts 2026-03-05 for mobile release." Converts relative dates to absolute.

reference

Pointers to external systems. "Pipeline bugs tracked in Linear INGEST project." "Oncall dashboard at grafana.internal/d/api-latency."

Frontmatter format: Each topic file has YAML frontmatter: name, description (used by Sonnet selector — be specific), type (user/feedback/project/reference). The description is what Sonnet reads to decide relevance — vague descriptions = never surfaced.

autoDream — Background Consolidation

Fires after ≥24h since last consolidation AND ≥5 sessions. Runs as a forked subagent via runForkedAgent().

Phase 1
Orient
Phase 2
Gather signal
Phase 3
Consolidate
Phase 4
Prune & index

Phase 1 — Orient

ls memory directory, read MEMORY.md, skim topic files, review daily logs if assistant-mode.

Phase 2 — Gather Signal

Priority 1: Daily logs. Priority 2: Drifted existing memories. Priority 3: Transcript grep with narrow terms (large JSONL — never read whole files).

Phase 3 — Consolidate

Write/update memory files. Merge signal into existing. Convert relative → absolute dates. Delete contradicted facts.

Phase 4 — Prune & Index

Update MEMORY.md pointers. Remove stale entries. Demote verbose entries. Add new memory pointers. Resolve contradictions.

Consolidation lock: .consolidate-lock file in memory dir containing PID. Stale guard at 60 minutes (prevents dead process holding lock forever). On failure, rollbackConsolidationLock() rewinds mtime so next session retries.

What NOT to Save

  • Code patterns, architecture, file paths — derivable from reading the project
  • Git history, recent changes — git log / git blame are authoritative
  • Debugging solutions — the fix is in the code, commit message has context
  • Anything already in CLAUDE.md files
  • Ephemeral task details — use tasks, not memory