dev: wait for port free in dev-reload; http: append startup audit line when writing pid#30
Open
mchzimm wants to merge 134 commits into
Open
dev: wait for port free in dev-reload; http: append startup audit line when writing pid#30mchzimm wants to merge 134 commits into
mchzimm wants to merge 134 commits into
Conversation
Day 5 exposes the Sentinel hook layer through 7 new CLI commands. The
code from Days 1-4 was inert (correct but unreachable); Day 5 makes it
the thing Claude Code actually invokes.
New commands:
engram intercept
Hook entry point. Reads JSON from stdin with 3s timeout, passes
through dispatchHook, writes JSON response to stdout. ALWAYS exits
0 — the process boundary is the last line of defense for the
"never block Claude Code" invariant. Even malformed input, missing
graph, or handler crash resolves to empty stdout (passthrough).
engram install-hook [--scope <local|project|user>] [--dry-run]
Surgically add engram's entries to a Claude Code settings.json.
Default scope is 'local' (.claude/settings.local.json, gitignored).
Preserves all existing non-engram hooks. Idempotent. Writes
atomically (temp file + rename) with timestamped backup
(settings.json.engram-backup-<ISO>.bak). --dry-run shows the diff
without writing.
engram uninstall-hook [--scope <s>]
Surgical removal. Only deletes entries whose command contains
"engram intercept". Cleans up empty event arrays and empty hooks
object. Backup before write.
engram hook-stats [--json]
Read .engram/hook-log.jsonl, summarize by event / tool / decision.
Shows estimated tokens saved based on PreToolUse Read deny count
(1200 tok/deny average). Human-readable text by default, JSON with
--json flag.
engram hook-preview <file>
Dry-run the Read handler for a specific file without installing.
Shows deny+reason (if confidence high), allow+context (landmines),
or passthrough with explanation. Perfect for debugging coverage.
engram hook-disable / hook-enable
Toggle .engram/hook-disabled kill switch. All handlers check this
flag; when set, everything falls through to passthrough without
uninstalling the settings.json entries.
New modules:
src/intercept/installer.ts - Pure data transforms:
buildEngramHookEntries,
installEngramHooks (idempotent),
uninstallEngramHooks (surgical),
isEngramHookEntry (detection),
formatInstallDiff (dry-run view).
Zero I/O — all reads/writes live
in cli.ts, tested independently.
src/intercept/stats.ts - summarizeHookLog +
formatStatsSummary. Pure
aggregation over HookLogEntry[].
Read-deny token savings: 1200
tok/deny estimate.
Modified:
src/intercept/dispatch.ts - Added decision logging for all
PreToolUse routes. Every Read /
Edit / Write / Bash invocation
logs {event, tool, path,
decision: deny|allow|passthrough}
to hook-log.jsonl after the
handler resolves. Logging errors
swallowed — never affects dispatch
result.
src/cli.ts - 7 new commander commands + helper
resolveSettingsPath(scope). All
install/uninstall writes are
atomic (temp + rename) with
timestamped backup.
Tests: +44 new (total 439, up from 395)
tests/intercept/installer.test.ts - 24 tests (idempotent install,
non-destructive, surgical
uninstall, immutability)
tests/intercept/stats.test.ts - 13 tests (summary correctness,
frozen results, formatting)
tests/intercept/cli-intercept.test.ts - 7 end-to-end subprocess tests
that actually spawn
'node dist/cli.js intercept'
and pipe JSON payloads. Auto-
builds dist/ via spawnSync npm
run build in beforeAll.
DOGFOOD VERIFIED in real shell:
$ node dist/cli.js init
$ node dist/cli.js hook-preview src/graph/query.ts
📋 Hook preview: /Users/nicholas/engram/src/graph/query.ts
Decision: DENY (Read would be replaced)
Summary (would be delivered to Claude):
[engram] Structural summary for src/graph/query.ts
Nodes: 10 | avg extraction confidence: 1.00
NODE queryGraph() [function] L80
NODE shortestPath() [function] L170
NODE renderFileStructure() [function] L412
... (~350 tokens instead of the ~4,000 token full file)
$ node dist/cli.js hook-stats
engram hook stats (3 invocations)
By event:
PreToolUse 3 (100.0%)
By tool:
Read 3
$ node dist/cli.js install-hook --dry-run
📌 engram install-hook (scope: local)
Target: /Users/nicholas/engram/.claude/settings.local.json
Changes:
+ PreToolUse: 0 → 1 entries
+ { matcher="Read|Edit|Write|Bash" command="engram intercept"}
+ PostToolUse: 0 → 1 entries
+ { matcher=".*" command="engram intercept"}
+ SessionStart: 0 → 1 entries
+ { command="engram intercept"}
+ UserPromptSubmit: 0 → 1 entries
+ { command="engram intercept"}
(dry-run — no changes written)
$ node dist/cli.js hook-disable
✅ engram hooks disabled for /Users/nicholas/engram
$ node dist/cli.js hook-preview src/graph/query.ts
Decision: PASSTHROUGH (Read would execute normally)
$ node dist/cli.js hook-enable
✅ engram hooks re-enabled for /Users/nicholas/engram
All 439 tests pass. tsc clean. Dogfooded on engram itself — Read
interception produces 11.1x token reduction for query.ts (4,000 →
350 tok).
Safety invariants preserved at the process boundary:
- engram intercept ALWAYS exits 0
- Stdin read with 3s hard timeout
- Input size cap: 1MB
- Any parse/dispatch error → empty stdout → passthrough
- install-hook backs up before writing
- uninstall-hook surgically removes only engram entries
Sentinel stack is now end-to-end functional. v0.3.0 is installable and
testable in a real Claude Code session.
Next: Day 6 — README rewrite with "context as infra" hero, CHANGELOG
entry, troubleshooting docs, opportunistic landmines rename in comments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…rsion bump
Ships the docs + metadata for v0.3.0 "Sentinel". No code behavior
changes — this is the packaging layer on top of Days 1-5.
README.md:
- New hero: "Context as infra for your AI coding tools"
- Replaced v0.2 quickstart with the Sentinel install flow:
engram init → engram install-hook
- New "The Problem" section explaining the v0.2 ceiling
(agent has to remember) and how v0.3 flips it (hook intercepts
at the tool-call boundary).
- New "How the Sentinel Layer Works" table — all 7 hook handlers
with their mechanism (deny+reason / allow+additionalContext /
pure observer) and purpose.
- 10 safety invariants listed verbatim.
- Sentinel command reference (7 new commands) as a separate
subsection, keeping the v0.1/v0.2 commands untouched for backcompat.
- Tests badge updated 132 → 439.
CHANGELOG.md:
- Full v0.3.0 entry with mechanism explanation, empirical
verification note, the 7 new CLI commands, the 7 new hook
handlers + projected savings per hook, infrastructure details,
content safety list, 10 safety invariants, test coverage
numbers, v0.3.1 deferrals (Grep, per-user thresholds, self-
tuning), and explicit "no migration needed" note.
package.json:
- version: 0.2.1 → 0.3.0
- description: updated to mention the hook interception layer
src/cli.ts:
- program.version("0.3.0")
- program.description() updated to match package.json
src/graph/query.ts:
- Opportunistic "regret buffer" → "landmines" rename in 4
comments. Internal API unchanged: mistakes() function,
list_mistakes MCP tool, kind: "mistake" schema all stable.
"Landmines" is the user-facing metaphor; "mistakes" is the
internal term. Memory feedback_engram_v0_3_sentinel_architecture.md
documents this split.
Verification:
$ npx tsc --noEmit → exit 0
$ npm run build → ✅ dist/cli.js 49.41 KB
$ node dist/cli.js --version → 0.3.0
$ npx vitest run → 439/439 passing (~1.5s)
Cumulative Sentinel stats through Day 6:
- 6 commits on v0.3.0-sentinel branch
- ~6,500 LOC (source + tests + docs)
- 439 tests (+225 from v0.2.1 baseline of 214)
- 7 hook handlers shipped (Read, Edit, Write, Bash, SessionStart,
UserPromptSubmit, PostToolUse)
- 7 new CLI commands (intercept, install-hook, uninstall-hook,
hook-stats, hook-preview, hook-disable, hook-enable)
- 10 safety invariants enforced at runtime
- Dogfood-verified: engram intercepts its own src/graph/query.ts
with an 11.1x token reduction
Next: Day 7 — benchmark on a real session, final pre-publish smoke
test, npm publish engramx@0.3.0, GitHub release.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Day 7 is the prep-for-ship day. Everything that can be done without
Nick's direct action (2FA, public release, account state) is done.
Everything that requires his explicit action is documented for him to
run at his own pace.
Added:
RELEASE-NOTES-v0.3.0.md — complete release notes with:
- Real measured Sentinel numbers on 4 engram files: 12,189 → 2,210
tokens (82% reduction, 75% hit rate) — measured not projected
- Real benchmark: 113,544 → 464 avg query tokens (244.7x vs full,
11.1x vs relevant files)
- All 7 new CLI commands documented
- All 7 new hook handlers with mechanism
- 10 safety invariants listed
- Zero-migration guarantee spelled out
- Step-by-step manual actions for Nick:
1. git tag v0.3.0 (DONE locally)
2. Review branch (git log --oneline main..v0.3.0-sentinel)
3. Merge to main (3 strategies documented)
4. Push origin + tag
5. npm publish (requires 2FA)
6. GitHub release
7. Install globally and try
8. Announce (launch posts template)
Local tag:
v0.3.0 created at 125af22 (Day 6 commit).
Not pushed — Nick's call.
Final verification:
$ npx tsc --noEmit → exit 0
$ npx vitest run → 439/439 passing (~1.5s)
$ npm run build → ✅ dist/cli.js 49.41 KB
$ node dist/cli.js --version → 0.3.0
$ npm pack --dry-run → engramx@0.3.0, 42.5 KB packed, 9 files
Cumulative Sentinel (Days 1-7):
Commits on branch: 7 (Day 1 → Day 2 → audit → Day 3 → Day 4 →
Day 5 → Day 6 → Day 7)
Total LOC delta: +6,505 source + tests + docs - 65 removed
Tests: 214 → 439 (+225 new, zero regressions)
Test suite time: ~1.5s
Hook handlers shipped: 7
CLI commands shipped: 7
Safety invariants enforced: 10
Empirically measured savings vs projection:
Projected (Day 0): -42,500 tok/session (80% reduction)
Measured on 4 engram files: -9,979 tok / 4 files (82% reduction)
Hit rate: 75% (projected 60%)
v0.3.0 is ready. The Sentinel ships.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
engram v0.3.0 "Sentinel" — the release that makes engram stop being a tool your agent queries and start being a Claude Code hook layer that intercepts Read/Edit/Write/Bash at the tool-call boundary. == What ships == Seven new hook handlers: - PreToolUse:Read deny+reason replaces file with ~300-tok summary - PreToolUse:Edit/Write allow+context with landmine warnings (never blocks) - PreToolUse:Bash strict parser + delegation to Read handler - SessionStart project brief injection (startup/clear/compact) - UserPromptSubmit keyword-gated pre-query injection - PostToolUse pure observer → hook-log.jsonl Seven new CLI commands: - engram intercept hook entry point (stdin → dispatch → stdout) - engram install-hook atomic, idempotent, backup, --dry-run - engram uninstall-hook surgical removal - engram hook-stats summarize .engram/hook-log.jsonl - engram hook-preview dry-run for a specific file - engram hook-disable/enable kill-switch toggle Ten safety invariants enforced at runtime: 1. Any handler error → passthrough (never block Claude Code) 2. 2s per-handler timeout 3. Kill switch respected by every handler 4. Atomic settings.json writes with timestamped backups 5. Never intercept outside project root 6. Never intercept binaries or secrets (.env/.pem/.key/credentials) 7. Never log user prompt content (privacy invariant, tested) 8. Never inject >8000 chars per hook response 9. Stale graph detection (file mtime > graph mtime → passthrough) 10. Partial-read bypass (offset/limit → passthrough) == Empirically measured savings == Sentinel vs raw Read on 4 real engram files: src/core.ts ~4,169 tok → DENY (13 nodes, 300 tok summary) src/graph/query.ts ~4,890 tok → DENY (10 nodes, 300 tok summary) src/intercept/dispatch.ts ~1,820 tok → DENY (5 nodes, 300 tok summary) src/intercept/handlers/read.ts ~1,310 tok → PASSTHROUGH (1 export, correctly below threshold) TOTAL: 12,189 → 2,210 tokens (-82%, 75% hit rate) == Migration == No migration needed. v0.3.0 is purely additive. All v0.2.1 commands, MCP tools, and schema are unchanged. Hook layer is opt-in via engram install-hook. == 8 commits on v0.3.0-sentinel branch == edce41b Day 1: scaffold intercept layer (safety, context, formatter) c94462c Day 2: Read handler + getFileContext + renderFileStructure 6e91706 audit: normalizePath try/catch + kindOrder cleanup + coverage TODO 94ad1fe Day 3: Edit/Write landmines, Bash cat delegation, dispatcher c8ecdab Day 4: SessionStart + UserPromptSubmit + PostToolUse + hook-log 05e8027 Day 5: CLI wiring — engram becomes installable and runnable 125af22 Day 6: README hero rewrite + CHANGELOG + landmines rename + version bump cddb1a0 Day 7: Release notes, real benchmark numbers, local v0.3.0 tag == Stats == +7,472 / -44 across 44 files 214 tests → 439 tests (+225) ~1.5s full suite time tsc clean dist/cli.js 50.6 KB, 42.5 KB packed, 9 files total Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Publishing the visual ecosystem walkthrough created for the v0.3.0
release. 11-page architecture diagram showing:
§01 The 4 hook events in order (SessionStart, UserPromptSubmit,
PreToolUse, PostToolUse) with fire timing, mechanism, and
per-hook token impact.
§02 The Read handler's 9-branch decision tree with a real deny+reason
JSON response alongside. Shows every passthrough branch (payload
shape, content safety, project resolution, kill switch, graph
coverage, staleness, confidence threshold) and the money path
where interception happens.
§03 The 6-layer ecosystem substrate — engram graph, MemPalace,
rule files, skills, agents, MCP servers — showing how engram
fits alongside the other persistence layers a Claude Code
session leans on.
§04 Real measured savings on 4 engram source files: 12,189 → 2,210
tokens (−82%), 75% hit rate. Reproducible with
engram hook-preview.
§05 Quick reference — the 8 CLI commands you'll actually use.
Design: Ink & Paper palette (editorial black-on-black with teal
accent), Fraunces variable serif for headings, Space Grotesk body,
JetBrains Mono code. Asymmetric grid layout with scroll-reveal
animations in HTML; forced-visible + page-break-hardened for PDF.
Added:
docs/engram-sentinel-ecosystem.pdf 1.02 MB, A3 landscape, 11 pages
docs/engram-sentinel-ecosystem.html 44 KB, single self-contained file,
Google Fonts CDN only, zero JS
frameworks
Changed:
README.md — new "Architecture Diagram" section after the quickstart,
linking to both PDF and HTML.
.gitignore — added .claude/settings.local.json (project-local Claude
Code hook install state, machine-specific).
Generated via Chrome headless with:
--headless=new
--force-prefers-reduced-motion (override for scroll-reveals)
--virtual-time-budget=15000 (let fonts + IO observer resolve)
--print-to-pdf-no-header
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three changes in one commit:
1. PALETTE REBUILD (from assets/banner.html, extracted verbatim):
- Background: #0a0a0c (warm near-black, not cool)
- Primary: rgb(217, 119, 6) — amber, the signature color
- Heading: Space Grotesk 700 compressed, -3px letter-spacing
- Body: Space Grotesk 300 (light, editorial)
- Code: JetBrains Mono
- Grid: 40px white at 0.015 opacity (identical to banner)
- Wash: amber radial blur top-right (banner convention)
- Highlight: rgba(74, 222, 128, 0.8) — ONLY for ✅ success markers
- Signature: engr<span class="a">a</span>m — the letter "a" in amber,
used in topbar, hero h1, footer version string
The previous versions (Ink & Paper / terminal-cyan) were
editorial dev-generic. This version is extracted verbatim from
the engram banner.html so it actually matches the brand.
2. SVG KNOWLEDGE GRAPH in the hero — amber nodes + amber edges,
central "Sentinel" node with gentle pulse animation. Mirrors the
graph visualization on the right side of banner.png. The central
node is labeled "Sentinel" with surrounding nodes for Read, Edit,
Write, Bash, Session, Prompt.
3. SPACING AUDIT — sections were "disconnected" because vh-based
padding clamps (clamp(4rem, 10vh, 8rem)) inflated to the max
value on A3 print, creating 14rem dead zones between sections.
Fixed by replacing all vh clamps with fixed rem values:
.hero: padding-block 3.5rem 2.5rem (was up to 14rem combined)
.section: padding-block 3rem (was up to 16rem combined)
.section-head: single-column stack instead of 4fr/8fr split with
dead left column
Print override: drop to 1.8rem, let sections flow across pages
instead of forcing page-break-inside:avoid
Result: 14 pages → 9 pages on A3 landscape.
4. ATTRIBUTION + BRAND LINEAGE:
- Topbar status area now includes: "a cirvgreen venture" link to
cirvgreen.com (amber "cirvgreen" wordmark)
- Footer gained a dedicated attribution row below the description:
"Created by Nicholas Ashkar · NickCirv — Part of the cirvgreen
ecosystem"
- Footer links row added cirvgreen.com alongside npm/github/release
Both are styled as ghost text with amber brand highlights.
5. SECTION CONNECTIVITY — added a subtle 2.4rem amber line at the
top-left of every section (via ::before pseudo) to create visual
rhythm and signal section boundaries without aggressive dividers.
6. GENERIC §03 CONTENT — the substrate section describes engram's
surfaces in generic terms (graph.db, rules files, git history,
Claude Code hook config, peer MCP servers, host AI client) with
no references to Nick's personal MemPalace/brain-os/vault setup.
Safe for public repo.
Files updated:
docs/engram-sentinel-ecosystem.html (61.9 KB, full rebuild)
docs/engram-sentinel-ecosystem.pdf (901 KB, A3 landscape, 11 pages)
Generated with:
chrome --headless=new --force-prefers-reduced-motion
--virtual-time-budget=15000 --print-to-pdf
(A3 landscape, 8mm margins)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ffold, EngramBench v0.1
The v0.3.1 release across 5 fronts:
1. TF-IDF keyword filter on UserPromptSubmit hook — kills the 76-node
noise bug where common-term prompts poisoned mature graphs. New
computeKeywordIDF helper in src/core.ts, IDF >= 1.386 threshold
(25% cutoff), top-N seed selection. 3 new tests.
2. engram memory-sync command + src/intercept/memory-md.ts — writes
structural facts into a marker-bounded block inside Anthropic's
native MEMORY.md. Complementary to Auto-Dream: prose memory stays
with Anthropic, structural graph stays with engram. Pure builder
+ upsert + atomic write. 16 new tests.
3. Cursor 1.7 beforeReadFile adapter scaffold
(src/intercept/cursor-adapter.ts + engram cursor-intercept CLI).
Wraps existing handleRead in Cursor's {permission, user_message}
shape. Experimental — wire-up lands in v0.3.2. 8 new tests.
4. EngramBench v0.1 — 10 structural task definitions in bench/tasks/
(find-caller, parent-class, import-graph, refactor-scope,
cross-file-flow, etc.) with scoring rubrics and expected tokens
per setup. bench/run.sh runner scaffold + results/TEMPLATE.csv.
5. Rebrand to "the structural code graph" — package description,
keywords, README hero.
466 tests passing (up from 442 in v0.3.0). Zero new runtime deps.
Schema unchanged. No breaking changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3 source bugs fixed (POSIX path normalization, CRLF YAML parsing, libuv assertion on Node 25 Windows), 5 test bugs fixed. New toPosixPath() single source of truth for graph path storage. isHardSystemPath now platform-aware (UNC, Windows, ProgramData). Post-init nudge suggests install-hook when Sentinel not yet wired. Experience Tiers section in README. Windows + fail-fast CI matrix. User manual (HTML, engram brand identity). 467 tests passing. Credit: ultrathink (shahe-dev) for root-cause analysis + patch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Four new capabilities that shift engram from tool to infrastructure: 1. PreCompact hook — re-injects god nodes + landmines before context compaction. First tool in the ecosystem whose context survives Claude Code's conversation compression. 2. CwdChanged hook — auto-switches project graph when user navigates to a different directory mid-session. 3. File watcher (engram watch) — incremental re-indexing via fs.watch. 300ms debounce, extension whitelist, ignored directories. Zero deps. 4. Mempalace bundle — SessionStart queries mcp-mempalace in parallel with graph queries and appends semantic findings to the brief. Graceful degradation if mempalace not installed. Also: edges.source_file index, transactional deleteBySourceFile, async execFile (not sync), per-instance debounce state. 486 tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The CLI version was hardcoded as "0.3.0" in commander's .version() call, causing `engram --version` to report the wrong version after bumps. Now reads from package.json via createRequire at runtime. Also bumps to 0.4.1 to publish the fix (0.4.0 tarball has stale version). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
AAA-designed HTML guide showing how memory tools, compression plugins, code review tools, and workflow managers integrate with engram. Includes real token savings numbers, 4 integration patterns (CLI, programmatic API, hook chain, SessionStart bundle), hook coexistence table, and per-tool-type guidance. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Real-time terminal HUD showing hook activity and token savings. Refreshes every second from hook-log.jsonl. Shows: - Total tokens saved (cumulative) - Hit rate with visual bar - Decision breakdown (intercepted/allowed/passthrough) - Top intercepted files with bar chart - Recent activity feed - Landmine warnings count Also aliased as `engram hud`. No external TUI dependencies. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New CLI command `engram hud-label` outputs JSON for Claude HUD's --extra-cmd protocol. Shows ⚡engram with token savings + visual hit rate bar (▰/▱). Runs in <20ms via hook-log parsing. Users can add to their Claude HUD: --extra-cmd="engram hud-label" States: ready → listening → savings + bar + percentage. Bar fills as hit rate climbs. Savings number grows over session. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
hud-label now searches parent directories for the nearest .engram/ instead of only checking $PWD. This fixes empty labels when Claude Code starts from a parent directory (e.g., /opt instead of /opt/crypto-bot). Uses the same walk-up pattern as the Sentinel hooks' findProjectRoot. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…uides - Hook table: 7 → 9 handlers (added PreCompact, CwdChanged) - SessionStart now mentions mempalace bundle - New Infrastructure commands section (watch, dashboard, hud-label) - Claude HUD integration example with visual bar - Docs section: links to user manual + integration guide - Removed duplicate install block Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…nstall-hook Users running `engram install-hook` now get the HUD visible in Claude Code automatically. Respects existing statusLine configs — only sets it when absent. Uninstall cleanly removes engram-owned statusLine entries. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Performance (CRITICAL): - renderFileStructure: replaced getAllNodes()/getAllEdges() with targeted SQL queries (getNodesByFile, getEdgesForNodes). Eliminates full table scan that silently timed out on large projects. - scoreNodes: replaced getAllNodes() with searchNodes() SQL seeding. O(matches) instead of O(all nodes) per query. - Edge ordering: sort by endpoint degree before slice(0,10) so god-node relationships appear first. Accuracy: - Go import detection: track import() block state, no longer fires on struct field tags like json:"name". - TS arrow function: require => in line, no longer matches const x = (someValue). - Comment exclusion: lines starting with // or * skipped before pattern matching. No more phantom nodes from commented-out code. - Confidence calibrated to 0.85 for regex extraction, reserving 1.0 for future tree-sitter. Correctness: - LIKE wildcards (% and _) escaped in searchNodes. - Removed phantom graphology dependency (in package.json, zero imports). 493 tests passing. Zero regressions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Context Spine spec: engram as central context routing layer integrating MemPalace, Context7, Obsidian into single rich injection packets. Target: 90% session-level token savings via provider cache in SQLite. - Advisory docs: external review of engram strategy (founder brief, strategic spec, Phase 0 plan, design skill spec). Includes analysis and fact-checking notes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds the SQLite cache layer that makes the Context Spine fast: - provider_cache table: stores resolved context from external providers (mempalace, context7, obsidian) with TTL-based staleness - CRUD: getCachedContext, setCachedContext, warmCache (bulk), pruneStaleCache, clearProviderCache, getCacheStats - ContextProvider interface: the contract all providers implement (resolve, warmup, isAvailable, tokenBudget, timeoutMs) - Provider priority ordering and type exports Per-Read cache lookup is <5ms (SQLite SELECT). Expensive provider resolution happens at SessionStart (warmup) or on first cache miss with a 200ms timeout and hint fallback. 17 new tests. 510 total, all passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Context Spine is now functional: Internal providers (Tier 1, always available, no cache): - engram:structure — structural summary from graph (existing, refactored) - engram:mistakes — known issues from mistake memory - engram:git — recent changes, churn rate, last author External providers (Tier 2, cached in SQLite): - mempalace — decisions/learnings from ChromaDB semantic memory - context7 — library docs for detected imports - obsidian — project notes from vault via REST API Resolver engine: - resolveRichPacket() — assembles from all providers in parallel - warmAllProviders() — bulk cache fill at SessionStart - Per-provider timeouts with graceful degradation - Priority ordering within 600-token total budget - Availability caching (check once per session) 10 new tests. 520 total, all passing. Build clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
From code review: - getEdgesForNodes: chunk IN clause at 400 IDs to stay under SQLite's 999 variable limit. Deduplicates across chunks. - rowToCachedContext: add ?? fallbacks on all fields to prevent null propagation from pre-migration rows. - warmCache: call save() after transaction commit, consistent with bulkUpsert. Prevents cache loss if process exits before close(). - getNodesByFile: add LIMIT 500 default to prevent unbounded materialization on generated files. 520 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Context Spine is now LIVE: Read handler integration: - After graph context passes all checks, attempts rich packet resolution from all available providers (structure, mistakes, git, mempalace, context7, obsidian) in parallel - 1.5s total timeout wraps the entire resolution — if exceeded, falls back to graph-only summary (existing v0.4 behavior) - Builds NodeContext with imports, test status, churn rate from graph - Rich packet served via existing deny+reason mechanism SessionStart warmup: - Calls warmAllProviders() fire-and-forget after building the brief - Pre-fills provider_cache table so subsequent Reads hit cache (<5ms) - Warmup failure is silent — never delays or blocks session start Provider availability: - Tier 1 (internal): 200ms availability check timeout - Tier 2 (external): 500ms availability check timeout - Results cached per-session (check once, reuse) - _resetAvailabilityCache() exposed for tests 520 tests passing. Build clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Removed all debug scaffolding (writeFileSync, ENGRAM_DEBUG guards)
- Fixed: global `engram` binary pointed to old npm install, not local
dev build. Future: version bump before npm publish.
- Enrichment header uses "[engram] Additional context" when structure
provider is excluded (enrichment-only mode)
The Context Spine is verified end-to-end:
echo '{"hook_event_name":"PreToolUse",...}' | engram intercept
→ Structure (8 nodes) + CHANGES (git provider) in one response
520 tests passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The biggest engram release since v0.3 Sentinel. Read interceptions now serve rich context assembled from 6 providers in parallel: structure, known issues, git changes, MemPalace decisions, Context7 library docs, and Obsidian project notes. One response replaces five tool calls. Includes 9 launch-critical fixes (perf + accuracy + correctness), provider cache in SQLite, parallel resolver with budget/timeout safety, and SessionStart cache warmup. 520 tests, all passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…keywords From repo audit: - package.json: added repository/homepage/bugs URLs (was missing — npm page had no source links) - package.json: updated description to Context Spine positioning - package.json: replaced "ast" keyword with "context-spine", "context-providers" - package.json: added CHANGELOG.md to files array - README: replaced stale v0.2-era roadmap with shipped/next sections - README: removed "v0.3 Sentinel" from Quickstart heading - Version bump to 0.5.1 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… test counts From repo audit — factual accuracy sweep: - README: import example uses "engramx" not "engram" (critical — wrong package name) - README: comparison table "AST extraction" → "Heuristic extraction" - README: "AST rebuild" → "graph rebuild" in git hooks section - docs/engram-user-manual.html: all 4 tree-sitter references replaced with "heuristic extraction" (was claiming tree-sitter which isn't implemented) - docs/engram-integration-guide.html: same tree-sitter fix + test count 486→520 - Removed orphaned RELEASE-NOTES-v0.3.0.md (stale, only file for one version) - Version bump to 0.5.2 We don't mislead users. Regex heuristics at 0.85 confidence is honest. Tree-sitter is planned, not shipped. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sql.js exposes getRowsModified() at runtime but the @types/sql.js type definitions don't include it, causing TS2339 on CI typecheck. Use SQLite's changes() function via db.exec() instead. Fixes all 4 CI matrix failures (ubuntu/windows × node 20/22). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
HTTP REST server (port 7337), tree-sitter AST provider (10 languages, confidence 1.0), LSP provider with diagnostic capture, auto-tuning from hook-log analysis, schema versioning with 6 migrations, CCS import/export, IDE adapters (Continue.dev, Cursor MDC, Zed context server, Aider), benchmark harness proving 88.1% token savings, stress test suite, component health HUD, and ECP spec v0.1. 35 new files, 579 tests (was 520), 8 providers (was 6), 5 IDE integrations. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
added 22 commits
May 3, 2026 05:24
…project-scoped nodes/queries (project_root, project_branch, memory_scope)
…mory-scope recording and retrieval
…s, providers); add graph stop handle to avoid leaks
…non-blocking DB saves; disable MEMORY.md auto writes via env guard
…t-tool, and query; add auto-memory module
…s and files on learn
… + merge-upsert; migrate canonical_id and merge duplicates; expose /api/graph/edges; fallback heatmap/hook-log to graph/session_log; background init on server start
…ram ui (PID-safe)
…rver via engram ui
… and explicit 'remember' auto-learn in UserPromptSubmit
…; inject SessionStart on resume; persist explicit remembers
…e when writing pid
5 tasks
There was a problem hiding this comment.
Pull request overview
This PR substantially expands engramx’s runtime/data model to support a single global DB with per-project scoping and “memory scopes” (project/global/entity), adds aggressive auto-memory ingestion paths, updates the dashboard to switch between scopes/projects, and also introduces a dev-reload helper script plus server startup auditing.
Changes:
- Add global DB + project scoping/memory scopes, plus canonical-id support for deterministic dedupe/merging.
- Add auto-memory ingestion (SessionStart, PostToolUse, explicit “remember…” prompts, query results) and new hook surface(s).
- Improve dev workflow (nodemon-based dev reload script) and expand the dashboard (scope/project selector, graph edges, token history).
Reviewed changes
Copilot reviewed 60 out of 61 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/memory-scope.test.ts | Adds coverage for storing/retrieving memories across scopes. |
| tests/intercept/spike-regression.test.ts | Updates expected intercept text prefixes to engramx. |
| tests/intercept/render-file-structure.test.ts | Updates expected structure summary header prefix. |
| tests/intercept/handlers/user-prompt.test.ts | Updates expected UserPromptSubmit context prefix. |
| tests/intercept/handlers/session-start.test.ts | Updates expected SessionStart brief prefix. |
| tests/intercept/handlers/read.test.ts | Updates expected Read deny reason prefix. |
| tests/intercept/handlers/pre-compact.test.ts | Updates expected PreCompact brief prefix. |
| tests/intercept/handlers/cwd-changed.test.ts | Updates expected cwd-changed context prefix. |
| tests/intercept/handlers/bash.test.ts | Updates expected bash deny reason prefix. |
| tests/intercept/formatter.test.ts | Updates expected formatter headers/prefixes. |
| tests/intercept/cli-intercept.test.ts | Updates expected CLI intercept output prefixes. |
| tests/fixtures/hook-payloads/README.md | Updates documented hook payload examples to engramx. |
| src/watcher.ts | Passes projectRoot/memoryScope through store operations; updates user-facing log strings. |
| src/tuner/index.ts | Renames built-in provider ids to engramx:*. |
| src/tuner/config.ts | Adds auto-memory config fields + defaults. |
| src/setup/wizard.ts | Updates IDE adapter instructions for engramx naming. |
| src/server/ui.ts | Adds dashboard scope/project selector, graph legend, tokens refresh + time-series, scoped API calls, SSE throttling. |
| src/server/ui-graph.ts | Adds edge rendering, scope-based node styling/shaping, consolidation logic, keyboard zoom, and stop-handle. |
| src/server/learn-ws.ts | Adds a minimal WebSocket ingestion protocol for learn(). |
| src/server/index.ts | Updates server startup stderr messages to engramx. |
| src/server/http.ts | Adds dashboard scope API, graph edges API, websocket upgrade path, and other scope-aware dashboard behaviors. |
| src/providers/types.ts | Updates provider docstrings and priority list to engramx:*. |
| src/providers/resolver.ts | Updates headers/provider ids and makes cache warming project-scoped. |
| src/providers/lsp.ts | Renames provider id to engramx:lsp. |
| src/providers/engram-structure.ts | Renames provider id/header docs to engramx:structure. |
| src/providers/engram-mistakes.ts | Renames provider id and scopes getNodesByFile calls. |
| src/providers/engram-git.ts | Renames provider id to engramx:git. |
| src/providers/ast.ts | Renames provider id/docs to engramx:ast. |
| src/miners/session-miner.ts | Switches session-mined node ids to canonical ids. |
| src/miners/pdf-miner.ts | Adds best-effort PDF text extraction helper (pdf-parse/pdftotext). |
| src/miners/linking-helpers.ts | Adds keyword/path/command extraction to infer linking candidates from free text. |
| src/miners/git-miner.ts | Updates timeout log prefix to engramx. |
| src/miners/conclusions-miner.ts | Adds conclusion + fragment node generation and rationale edges using canonical ids. |
| src/intercept/memory-md.ts | Renames markers to engramx:* and adds write-to-arbitrary-path helper + opt-out env var. |
| src/intercept/handlers/user-prompt.ts | Adds explicit remember→learn path and updates injected header prefix. |
| src/intercept/handlers/session-start.ts | Injects brief on resume; triggers auto-memory on session start; updates header prefix. |
| src/intercept/handlers/read.ts | Updates provider ids and makes node/edge queries project-scoped. |
| src/intercept/handlers/pre-compact.ts | Updates brief header prefix. |
| src/intercept/handlers/post-tool.ts | Triggers auto-memory ingestion on Read/Edit/Write tool events. |
| src/intercept/handlers/mistake-guard.ts | Makes mistake queries project-scoped; updates provider reference string. |
| src/intercept/handlers/cwd-changed.ts | Updates injected header prefix. |
| src/intercept/handlers/assistant-message.ts | Adds opt-in hook to persist assistant content as memory (fire-and-forget). |
| src/intercept/dispatch.ts | Adds AssistantMessage dispatch; improves cost path resolution; records session token stats. |
| src/intercept/auto-memory.ts | Adds aggressive auto-memory ingestion + per-scope dedupe hash stats. |
| src/intelligence/token-tracker.ts | Adds project-scoped token stats and session time-series log. |
| src/integrations/pi.ts | Adds a minimal Node 20+ HTTP client for /hook, /learn, /query. |
| src/hooks.ts | Updates hook output prefix to engramx. |
| src/graph/store.ts | Adds project scoping columns, canonical-id merge upsert, async save, and scoped query methods. |
| src/graph/query.ts | Adds projectRoot scoping to graph traversal + updates structure summary header prefix. |
| src/graph/canonical.ts | Adds deterministic canonical id generation helper. |
| src/doctor/report.ts | Updates AST reinstall guidance to engramx. |
| src/db/migrate.ts | Bumps schema version + adds migrations for project scoping and canonical ids. |
| src/core.ts | Switches to a single global DB; adds project stat key helper; adds scoped query; expands learn() with conclusions/linking; adds token instrumentation. |
| src/cli.ts | Adds learn --scope; expands memory-sync to support scopes and new output locations; updates error prefixes. |
| src/autogen.ts | Renames autogen markers and error prefix to engramx. |
| scripts/dev-reload.sh | Adds dev reload helper that kills listeners, rebuilds, and waits for port to free before restart. |
| package.json | Adds dev:reload and nodemon devDependency. |
| package-lock.json | Locks nodemon and transitive deps. |
| examples/pi-client.js | Adds an example client for the HTTP server /hook and /learn. |
| docs/operations/dev-reload-port-audit.md | Documents dev-reload port waiting and startup audit behavior + test steps. |
| docs/integrations/pi.md | Documents external integration via the local HTTP server and example client. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+114
to
+121
| // If export fails for any reason, fall back to a best-effort | ||
| // synchronous write attempt (rare). Swallow errors to avoid | ||
| // crashing callers. | ||
| try { | ||
| writeFileSync(this.dbPath, Buffer.from([])); | ||
| } catch { | ||
| /* swallow */ | ||
| } |
Comment on lines
+175
to
200
| upsertNode(node: GraphNode, defaults?: { projectRoot?: string; projectBranch?: string; memoryScope?: string }): void { | ||
| const projectRoot = (node as any).projectRoot ?? defaults?.projectRoot ?? ""; | ||
| const projectBranch = (node as any).projectBranch ?? defaults?.projectBranch ?? null; | ||
| const memoryScope = (node as any).memoryScope ?? defaults?.memoryScope ?? null; | ||
|
|
||
| this.db.run( | ||
| `INSERT OR REPLACE INTO nodes (id, label, kind, source_file, source_location, confidence, confidence_score, last_verified, query_count, metadata, valid_until, invalidated_by_commit) | ||
| VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, | ||
| `INSERT OR REPLACE INTO nodes (id, label, kind, source_file, source_location, confidence, confidence_score, last_verified, query_count, metadata, valid_until, invalidated_by_commit, project_root, project_branch, memory_scope) | ||
| VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, | ||
| [ | ||
| node.id, | ||
| node.label, | ||
| node.kind, | ||
| node.sourceFile, | ||
| node.sourceLocation, | ||
| node.confidence, | ||
| node.confidenceScore, | ||
| node.lastVerified, | ||
| node.queryCount, | ||
| JSON.stringify(node.metadata), | ||
| node.validUntil ?? null, | ||
| node.invalidatedByCommit ?? null, | ||
| projectRoot, | ||
| projectBranch, | ||
| memoryScope, | ||
| ] | ||
| ); |
| const store = await getStore(root); | ||
| try { | ||
| const allNodes = store.getAllNodes(); | ||
| const allNodes = store.getAllNodes(projectRoot); |
Comment on lines
575
to
+600
| export async function learn( | ||
| projectRoot: string, | ||
| text: string, | ||
| sourceLabel = "manual" | ||
| sourceLabel = "manual", | ||
| memoryScope: string = "project" | ||
| ): Promise<{ nodesAdded: number }> { | ||
| const { nodes, edges } = learnFromSession(text, sourceLabel); | ||
| if (nodes.length === 0 && edges.length === 0) return { nodesAdded: 0 }; | ||
| // Primary session mining (decisions/mistakes/patterns) | ||
| const sessionResult = learnFromSession(text, sourceLabel); | ||
| const conclusionResult = generateConclusionNodes(text, sourceLabel); | ||
|
|
||
| const combinedNodes = [...sessionResult.nodes, ...conclusionResult.nodes]; | ||
| const combinedEdges = [...sessionResult.edges, ...conclusionResult.edges]; | ||
|
|
||
| if (combinedNodes.length === 0 && combinedEdges.length === 0) return { nodesAdded: 0 }; | ||
|
|
||
| const store = await getStore(projectRoot); | ||
| try { | ||
| store.bulkUpsert(nodes, edges); | ||
| // Bulk upsert nodes + edges (project-scoped) | ||
| store.bulkUpsert(combinedNodes, combinedEdges, projectRoot, undefined, memoryScope); | ||
|
|
||
| // Post-insert: create linking edges from conclusion nodes to existing | ||
| // graph nodes by simple keyword overlap. This helps surface relations | ||
| // between learned conclusions/fragments and code entities/files. | ||
| const now = Date.now(); | ||
| const allNodes = store.getAllNodes(projectRoot); | ||
|
|
Comment on lines
280
to
286
| const root = pathResolve(opts.project); | ||
| if (!existsSync(join(root, ".engram", "graph.db"))) { | ||
| console.error( | ||
| `engram: no graph found at ${root}. Run 'engram init' first.` | ||
| `engramx: no graph found at ${root}. Run 'engram init' first.` | ||
| ); | ||
| process.exit(1); | ||
| } |
Comment on lines
+166
to
+174
| // Explicit memory requests: if the user explicitly asks the agent to | ||
| // remember/save/store this prompt, persist it to the project's memory. | ||
| // Fire-and-forget so we don't delay the hook response. We bypass the | ||
| // autoMemoryEnabled flag for explicit user requests. | ||
| try { | ||
| const explicitRemember = /^\s*(?:remember|save|store|dont forget|don't forget|note)\b/i; | ||
| if (explicitRemember.test(prompt)) { | ||
| void learn(projectRoot, prompt, "user:remember", "project").catch(() => {}); | ||
| } |
Comment on lines
+741
to
+747
| window.addEventListener('keydown', (ev) => { | ||
| if (ev.key === '-' || ev.key === '_') { | ||
| keyboardZoom(0.9); | ||
| } else if (ev.key === '=' || ev.key === '+' ) { | ||
| keyboardZoom(1.1); | ||
| } | ||
| }); |
Comment on lines
+213
to
+218
| // Compute canonical ids for existing nodes and backfill | ||
| try { | ||
| // Import computeCanonicalId dynamically so migrations are self-contained | ||
| const { computeCanonicalId } = require("../graph/canonical.js"); | ||
| const res = db.exec("SELECT id, label, kind, memory_scope, project_root FROM nodes"); | ||
| const rows = (res[0] && res[0].values) ? res[0].values : []; |
Comment on lines
+23
to
34
| // Global DB config: single database for all projects | ||
| const GLOBAL_DB_DIR = process.env.ENGRAM_GLOBAL_DB_DIR || join(homedir(), ".engramx"); | ||
| const GLOBAL_DB_FILE = process.env.ENGRAM_GLOBAL_DB_FILE || "memory.db"; | ||
|
|
||
| export function getGlobalDbPath(): string { | ||
| return process.env.ENGRAM_GLOBAL_DB_PATH || join(GLOBAL_DB_DIR, GLOBAL_DB_FILE); | ||
| } | ||
|
|
||
| export function getDbPath(_projectRoot: string): string { | ||
| // Backwards-compatible alias: always use the single global DB. | ||
| return getGlobalDbPath(); | ||
| } |
Comment on lines
+738
to
+742
| const projects = await listKnownProjects(store); | ||
| const scopes = [ | ||
| { id: "accumulative", label: "ACCUMMULATIVE MEMORIES" }, | ||
| { id: "global", label: "GLOBAL MEMORIES" }, | ||
| { id: "personal", label: "PERSONAL MEMORIES" }, |
added 2 commits
May 8, 2026 04:07
…pear in dashboard
…ter learn to populate file nodes
Author
|
Additional fixes included in this branch:
I also retroactively added missing project_root entries for existing projects in your local DB (e.g., /Users/max/prjs/worlds.ai.rust) so the dashboard now lists them. If you prefer a different approach (no background init), I can revert that part and instead expose a UI hint to run 🔍 Scanning codebase... 📊 Token savings: 111.6x fewer tokens vs relevant files (211.2x vs full corpus) ✅ Ready. Your AI now has persistent memory. 💡 Next step: engram install-hook — enables automatic Read interception (82% token savings) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Files changed:
Testing steps are in docs/operations/dev-reload-port-audit.md.
Follow-ups / todos are listed in the docs file and tracked in issue #31.