feat(store): persistent SQLite event log + JSON-file artifact store#232
Merged
Conversation
Lands the store-only slice of the persistent-backends group (#42, #174, #223). Two new protocol-conformant backends + an EventLog lifecycle contract refinement; zero behavior change for existing in-memory users. #223 — SqliteEventLog + shared _sqlite_base.py - store/_sqlite_base.py: connection helper (WAL, foreign_keys=ON, parent dir auto-create) + apply_migrations() driven by a _contextweaver_schema_version table. The other 3 SQLite stores in epic #174 reuse this. - store/sqlite_event_log.py: implements every EventLog protocol method against a single-process SQLite file. Append-only writes via append() (duplicate ids raise DuplicateItemError); insertion order preserved by an auto-increment ordinal column; indexes on kind and parent_id. Round-trips every ContextItem field (metadata + artifact_ref via JSON columns). - [sqlite] extras placeholder in pyproject.toml (stdlib sqlite3 only; placeholder exists so a future aiosqlite variant can opt in without changing the install surface). #42 — JsonFileArtifactStore - store/json_file_artifacts.py: filesystem-backed ArtifactStore. Layout {base_dir}/{handle}.data + {base_dir}/{handle}.json. Auto- creates base_dir; rejects handles containing path separators / .. / null bytes. Re-instantiating recovers refs from existing metadata files. - store/artifacts.py: extracts the drilldown selector dispatch (head / lines / json_keys / rows) to a module-private _apply_selector() helper so both backends share one implementation. Behaviour preserved bit-for-bit; existing tests untouched. EventLog lifecycle (Mode B addition for clean SQLite cleanup) - store/protocols.py: EventLog protocol now requires close(), __enter__, __exit__. - store/event_log.py: InMemoryEventLog.close() is a no-op; existing callers continue to work without changes. `with SqliteEventLog(path) as log:` is the recommended idiom for the new backend. Tests: 63 new (32 SqliteEventLog, 29 JsonFileArtifactStore, 2 in-memory lifecycle). New modules at 87–100% coverage. Full suite 1011 passed, 8 skipped. Verification ruff format --check src/ tests/ examples/ scripts/ - clean ruff check src/ tests/ examples/ scripts/ - clean mypy src/ - 0 issues / 67 files pytest --cov=contextweaver -q - 1011 passed, 8 skipped make example, make demo, make scorecard-check - clean make llms-check - up to date Closes #42 Closes #223 Refs #174 https://claude.ai/code/session_01ADTmGUqM66tqnGMqrefy4e
There was a problem hiding this comment.
Pull request overview
Adds persistent store backends for the store layer while preserving the existing in-memory implementations.
Changes:
- Adds
SqliteEventLogwith shared SQLite connection/migration scaffolding. - Adds
JsonFileArtifactStoreand shares artifact drilldown selector logic. - Extends
EventLoglifecycle support withclose()and context manager methods, plus tests and docs/changelog updates.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
src/contextweaver/store/_sqlite_base.py |
Adds reusable SQLite connection and migration helpers. |
src/contextweaver/store/sqlite_event_log.py |
Adds SQLite-backed EventLog implementation. |
src/contextweaver/store/json_file_artifacts.py |
Adds filesystem-backed artifact persistence. |
src/contextweaver/store/artifacts.py |
Extracts shared drilldown selector helper. |
src/contextweaver/store/event_log.py |
Adds no-op lifecycle methods to in-memory event log. |
src/contextweaver/store/protocols.py |
Extends EventLog protocol with lifecycle methods. |
src/contextweaver/store/__init__.py |
Re-exports new store backends. |
tests/test_store_sqlite_event_log.py |
Adds SQLite event log coverage. |
tests/test_store_json_file_artifacts.py |
Adds JSON-file artifact store coverage. |
tests/test_store_event_log.py |
Adds in-memory event log lifecycle tests. |
pyproject.toml |
Adds placeholder [sqlite] optional extra. |
CHANGELOG.md |
Documents new persistent stores and lifecycle contract. |
AGENTS.md |
Updates agent-facing module map for new store backends. |
Benchmark delta (vs
|
| size | recall@k (head Δ vs base) | MRR (head Δ vs base) | p99 (ms) |
|---|---|---|---|
| 50 | ✅ 0.5649 (+0.0000) | ✅ 0.4978 (+0.0000) | ✅ 0.453 (base 0.463) |
| 83 | ✅ 0.3825 (+0.0000) | ✅ 0.3242 (+0.0000) | ✅ 0.624 (base 0.876) |
| 1000 | ✅ 0.1475 (+0.0000) | ✅ 0.1456 (+0.0000) | ✅ 30.611 (base 31.897) |
Per-backend × per-size matrix
| backend | size | recall@k (Δ) | MRR (Δ) | p99 (ms) |
|---|---|---|---|---|
| bm25 | 100 | ✅ 0.3825 (+0.0000) | ✅ 0.3399 (+0.0000) | ✅ 5.797 (base 5.642) |
| bm25 | 500 | ✅ 0.2250 (+0.0000) | ✅ 0.2165 (+0.0000) | ✅ 28.868 (base 27.538) |
| bm25 | 1000 | ✅ 0.1575 (+0.0000) | ✅ 0.1525 (+0.0000) | ✅ 78.840 (base 78.368) |
| fuzzy | 100 | ✅ 0.0000 (+0.0000) | ✅ 0.0000 (+0.0000) | ✅ 0.000 (base 0.000) |
| fuzzy | 500 | ✅ 0.0000 (+0.0000) | ✅ 0.0000 (+0.0000) | ✅ 0.000 (base 0.000) |
| fuzzy | 1000 | ✅ 0.0000 (+0.0000) | ✅ 0.0000 (+0.0000) | ✅ 0.000 (base 0.000) |
| tfidf | 100 | ✅ 0.3825 (+0.0000) | ✅ 0.3220 (+0.0000) | ✅ 0.911 (base 0.872) |
| tfidf | 500 | ✅ 0.2325 (+0.0000) | ✅ 0.2314 (+0.0000) | ✅ 8.230 (base 8.660) |
| tfidf | 1000 | ✅ 0.1475 (+0.0000) | ✅ 0.1456 (+0.0000) | ✅ 31.190 (base 30.071) |
Context pipeline (per scenario)
| scenario | tokens | dropped | dedup |
|---|---|---|---|
| large_catalog | 1514 (base 1514, Δ+0) | 0 (base 0, Δ+0) | 0 (base 0, Δ+0) |
| long_conversation | 2548 (base 2548, Δ+0) | 0 (base 0, Δ+0) | 0 (base 0, Δ+0) |
| short_conversation | 496 (base 496, Δ+0) | 0 (base 0, Δ+0) | 0 (base 0, Δ+0) |
| stress_conversation | 6651 (base 6651, Δ+0) | 7 (base 7, Δ+0) | 4 (base 4, Δ+0) |
Numbers come from make benchmark / make benchmark-matrix.
Latency is hardware-dependent — treat the markers as a rough guide.
See benchmarks/scorecard.md for the full picture.
Addresses all 4 Copilot review comments on the persistent-store PR. 1. **SqliteEventLog.query filter order mismatch with InMemoryEventLog** (Copilot #1). The SQL path applied `kinds` before `since`, while the in-memory path applies `since` to the full insertion-ordered log *before* filtering by kind. On mixed-kind logs this gave different results for the same `(kinds, since, limit)` triple. Switched to pull-all + slice-by-since + filter-by-kind + slice-by-limit, matching the in-memory semantics byte-for-byte. The existing test_query_combined_filters expected the buggy ordering and was updated; a new test_query_kinds_since_matches_in_memory_semantics pins parity across four representative (kinds, since, limit) cases against an InMemoryEventLog mirror. 2. **JsonFileArtifactStore path traversal** (Copilot #2). Handle validation was only enforced inside `put()`, leaving `get`, `ref`, `exists`, `delete`, `metadata`, and `drilldown` vulnerable to `..`, path separators, and absolute paths (read / delete outside `base_dir`). Centralised the validation inside `_meta_path` and `_data_path`, so every public method that resolves a handle is now safe. Parametrised regression test `test_read_path_methods_reject_unsafe_handles` covers all five read / mutate methods × six unsafe-handle shapes; a separate `test_drilldown_rejects_unsafe_handle` covers the drilldown path. 3. **Bare RuntimeError on store-closed** (Copilot #3). AGENTS.md requires public-facing errors to come from `contextweaver.exceptions`. Added `StoreClosedError(ContextWeaverError)` to the exceptions module and replaced the `RuntimeError` raised by `_require_conn` with it. The lifecycle docstring on `close()` now references the new exception. `test_use_after_close_raises` updated to assert both the specific exception and that it remains catchable as `ContextWeaverError`. 4. **list_refs crashes on wrong-shape JSON** (Copilot #4). `ArtifactRef.from_dict()` raises `TypeError` (not `ValueError`) when given a non-mapping JSON value such as `[]`, `null`, or a bare string. `list_refs` only caught `json.JSONDecodeError | KeyError | ValueError`, so a wrong-shape file crashed the whole listing. Added `TypeError` to the except clause and documented the failure-mode taxonomy on the method. `test_list_refs_skips_non_mapping_json` covers all three shapes. Verification: - ruff format/lint clean - mypy strict — 0 issues / 67 source files - pytest -q — 1047 passed, 5 skipped (+36 new tests over baseline) - make example + make demo clean - make scorecard-check + make llms-check clean
…7Pd1o # Conflicts: # CHANGELOG.md # pyproject.toml
Rich outputs UTF-8 box-drawing characters that fail to decode with the default cp1252 codec on Windows. Adding encoding='utf-8' to subprocess.run matches the PYTHONIOENCODING=utf-8 already set in the subprocess environment.
- Re-export StoreClosedError, SqliteEventLog, JsonFileArtifactStore from contextweaver/__init__.py and __all__ - Use contextlib.suppress for rollback safety in _sqlite_base.py - Clarify delete() docstring in JsonFileArtifactStore
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands the store-only slice of the persistent-backends group (#42, #174,
#223). Two new protocol-conformant backends + an EventLog lifecycle
contract refinement; zero behavior change for existing in-memory users.
Why
The context engine's event log and artifact store are in-memory only —
session data vanishes on process exit. Agents that persist across
restarts or need auditable history require durable backends. This PR
delivers the first two: a SQLite event log and a filesystem artifact
store, both conforming to the existing protocol interfaces.
What changed
SqliteEventLog + shared
_sqlite_base.py(#174, #223)store/_sqlite_base.py: connection helper (WAL,foreign_keys=ON,parent-dir auto-create) +
apply_migrations()driven by a_contextweaver_schema_versiontable. Designed for reuse by theremaining SQLite stores in epic [epic] SQLite persistent stores #174.
store/sqlite_event_log.py: implements everyEventLogprotocolmethod against a single-process SQLite file. Append-only writes via
append()(duplicate ids →DuplicateItemError); insertion orderpreserved by an auto-increment
ordinalcolumn; indexes onkindand
parent_id. Round-trips everyContextItemfield including JSONmetadataand nestedArtifactRef.[sqlite]extras placeholder inpyproject.toml(stdlibsqlite3only; placeholder for a future
aiosqliteasync variant).JsonFileArtifactStore (#42)
store/json_file_artifacts.py: filesystem-backedArtifactStore.Layout:
{base_dir}/{handle}.data+{base_dir}/{handle}.json.Auto-creates
base_dir; rejects handles containing path separators,..,., or null bytes. Re-instantiating against an existingdirectory recovers refs from metadata files.
store/artifacts.py: extracts drilldown selector dispatch (head/lines/json_keys/rows) to a shared_apply_selector()helper so both backends produce identical output.
EventLog lifecycle
store/protocols.py:EventLogprotocol now requiresclose(),__enter__,__exit__.store/event_log.py:InMemoryEventLog.close()is a no-op; existingcallers unaffected.
with SqliteEventLog(path) as log:is therecommended idiom.
Review-driven fixes
SqliteEventLog.query():sinceapplied to the full insertion-orderedlog before
kindsfilter, matchingInMemoryEventLogsemantics.JsonFileArtifactStore: path-traversal validation moved into_meta_path/_data_pathso every public method (not justput)rejects unsafe handles.
StoreClosedError(new exception inexceptions.py): replaces bareRuntimeErrorfor use-after-close, keeping theContextWeaverErrorhierarchy consistent per
AGENTS.md.JsonFileArtifactStore.list_refs(): catchesTypeErrorfromArtifactRef.from_dictwhen a.jsonfile is valid JSON but not amapping.
__init__.py: re-exportsSqliteEventLog,JsonFileArtifactStore, andStoreClosedError._sqlite_base.py: rollback inapply_migrationsusescontextlib.suppressfor safety if the connection is lost.Non-goals (out of scope)
aiosqlite).ArtifactStore,EpisodicStore,FactStore(futureslices of [epic] SQLite persistent stores #174).
Test plan
65 new tests: 32 SqliteEventLog, 29 JsonFileArtifactStore, 2 in-memory
lifecycle, 2 parity tests (SQLite vs in-memory query semantics).
Risks
Low. Both backends are opt-in (imported explicitly), single-process, and
have no effect on the default in-memory code path. The
EventLogprotocol addition (
close/ context manager) is backwards-compatiblesince the in-memory implementation's
close()is a no-op.Closes #42
Closes #223
Refs #174